Methods and systems for navigating a vehicle including a novel fiducial marker system

ABSTRACT

Methods and systems for navigating a vehicle along a surface employ a scanner to scan a light beam over the surface; employ light reflected by one or more fiducial markers on the surface onto pixels of a receiver to determine a spatial arrangement of the fiducial markers on the surface; and compare the spatial arrangement of the fiducial markers with a predetermined map of the fiducial markers to determine a location of the vehicle.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Utility patent application based on previouslyfiled U.S. Provisional Patent Application U.S. Ser. No. 62/707,194,filed on Oct. 19, 2017, the benefit of the filing date of which ishereby claimed under 35 U.S.C. § 119(e) and which is furtherincorporated in entirety by reference.

TECHNICAL FIELD

The present invention relates generally to a navigation or imagingsystem and to methods of making and using the navigation or imagingsystem. The present invention is also directed a navigation system thatuses fiducial markers in a surface to identify a location of thevehicle.

BACKGROUND

The objective of a high-fidelity 3D motion capture system is toaccurately observe and track objects and structures in the real world.Our world is a three-dimensional space where all observable structures,objects and shapes have spatial geometries. To fully describe thesegeometries actually takes 6 dimensions, or 6 degrees of freedom (DoF).For example, a small projectile may be tracked at a position in space inthree Cartesian coordinates (x, y, & z). To describe the projectile'sorientation at that position requires three additional dimensions, oftendescribed in navigational terms as rotational dimensions, such as roll,pitch and yaw. (In unmanned aerial vehicles (UAVs) these rotationsaround the longitudinal, horizontal and vertical axis respectively arethe key control and stability parameters determining the flightdynamics.)

Typically there is at least some motion between the observer (i.e. theviewer, the camera, or the sensor) and the observed objects or surfaces.From the observer's perspective an object's motion results in atrajectory (a path through space followed in time) with an instantaneousposition, velocity, curvature and acceleration, where each of thesequantities are functions which express the dimensions as a function oftime.

Sometimes moving objects follow simple trajectories that can be fullymodeled by elementary physics (for example, satellites, billiard balls,and ballistic projectiles). More often, things are not quite so simple.As an example, in robotics, when tracking grippers in robot arms, theobservational system itself may be subject to noisy random or evenchaotic multi-dimensional disturbances (rotations, translations,vibrations, or the like) resulting in compound measurement errors whichcan be considered a form of data flow entropy that furthermorecomplicates sensor data fusion at a system level.

Furthermore, objects and surfaces that are to be tracked may benon-rigid changing shapes such as, for example, a deformable elasticstructure or an organic surface such as a human face. Tracking such adeformable surface or object with high accuracy favors methods with highrelative local accuracy. This may, for example, take the form of ahighly accurate measurement of the relative 3D displacements betweenpoints in a surface mesh. Once detected, such deformable objects andtheir changing 3D shapes and trajectories need to be positioned withinsome kind of global context (i.e., a stable reference system.) Theselocal object measurements should be converted to a globalspatial-temporal coordinates without a loss of accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an embodiment of an exemplary environment in which variousembodiments of the invention may be implemented;

FIG. 2 illustrates an embodiment of an exemplary mobile computer thatmay be included in a system such as that shown in FIG. 1;

FIG. 3 shows an embodiment of an exemplary network computer that may beincluded in a system such as that shown in FIG. 1;

FIG. 4 shown the effect of camera vibration on motion blur of images;

FIG. 5A illustrates one embodiment of scanning point illumination;

FIG. 5B illustrates one embodiment of strobed line illumination;

FIG. 6 illustrates one embodiment of a scanning illumination system forobserving fiducial makers in a surface;

FIG. 7A is a top view illustrating a vehicle utilizing one embodiment ofsystem for scanning a road and navigating the vehicle;

FIG. 7B is a side view of the vehicle and embodiment of FIG. 7B;

FIG. 8A is a side perspective view of one embodiment of a vehicleutilizing a navigation system to operate an active suspension betweenthe cabin portion and undercarriage of the vehicle;

FIG. 8B is a side perspective view of another embodiment of a vehiclewith a cabin portion attached to an undercarriage;

FIG. 9 is a top view of a vehicle and a UAV utilizing a navigationsystem to detect IEDs or other objects;

FIG. 10 illustrates a diagram of light beams for fast foveatedperception;

FIG. 11 illustrates an area of interest located on pixels of a camera;

FIG. 12 illustrates operation of a system that masks out raindrops orsnow;

FIG. 13A illustrates a headlight arrangement with a light source andphosphor;

FIG. 13B illustrates light emitted by the headlight arrangement of FIG.13A over time;

FIG. 13C illustrates a narrowband filtered pixel in an array thatreceives laser blue light and triggers adjacent red (R), green (G), andblue (B) pixels;

FIG. 14 illustrates an arrangement for alternating stereo crossillumination;

FIG. 15 illustrates operation of alternating retroreflective fiducialdetection cycles;

FIG. 16A illustrates an arrangement for observing a common field of viewusing two cameras and light sources;

FIG. 16B illustrates an arrangement for observing a common field of viewusing two cameras and one light source;

FIG. 17 illustrates a head mounted display which tracks the eye gaze ofthe user;

FIG. 18 illustrates an arrangement of colored fiducial markers in aroad;

FIG. 19 illustrates an embodiment of a camera or sensor utilizing“twitchy pixels”;

FIG. 20A-C illustrate fiducial markers in a road surface;

FIG. 21A illustrates a top view of a section of road surface, in whichat random places retro-reflective fiducial markers have been placed;

FIG. 21B illustrates a side view of the illumination of a fiducialmarker and the reflection of light back to a camera; and

FIG. 22 illustrates an embodiment where two beams (B1 and B2) illuminatethe road surface below and cause successive retro reflections from twolines.

DETAILED DESCRIPTION OF THE INVENTION

Various embodiments now will be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific embodiments by which theinvention may be practiced. The embodiments may, however, be embodied inmany different forms and should not be construed as limited to theembodiments set forth herein; rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the embodiments to those skilled in the art. Amongother things, the various embodiments may be methods, systems, media, ordevices. Accordingly, the various embodiments may take the form of anentirely hardware embodiment, an entirely software embodiment, or anembodiment combining software and hardware aspects. The followingdetailed description is, therefore, not to be taken in a limiting sense.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may.Furthermore, the phrase “in another embodiment” as used herein does notnecessarily refer to a different embodiment, although it may. Thus, asdescribed below, various embodiments of the invention may be readilycombined, without departing from the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” include pluralreferences. The meaning of “in” includes “in” and “on.”

As used herein, the terms “photon beam,” “light beam,” “electromagneticbeam,” “image beam,” or “beam” refer to a somewhat localized (in timeand space) beam or bundle of photons or electromagnetic (EM) waves ofvarious frequencies or wavelengths within the EM spectrum. An outgoinglight beam is a beam that is transmitted by various ones of the variousembodiments disclosed herein. An incoming light beam is a beam that isdetected by various ones of the various embodiments disclosed herein.

As used herein, the terms “light source,” “photon source,” or “source”refer to various devices that are capable of emitting, providing,transmitting, or generating one or more photons or EM waves of one ormore wavelengths or frequencies within the EM spectrum. A light orphoton source may transmit one or more outgoing light beams. A photonsource may be a laser, a light emitting diode (LED), an organic lightemitting diode (OLED), a light bulb, or the like. A photon source maygenerate photons via stimulated emissions of atoms or molecules, anincandescent process, or various other mechanism that generates an EMwave or one or more photons. A photon source may provide continuous orpulsed outgoing light beams of a predetermined frequency, or range offrequencies. The outgoing light beams may be coherent light beams. Thephotons emitted by a light source may be of various wavelengths orfrequencies.

As used herein, the terms “camera”, “receiver,” “photon receiver,”“photon detector,” “light detector,” “detector,” “photon sensor,” “lightsensor,” or “sensor” refer to various devices that are sensitive to thepresence of one or more photons of one or more wavelengths orfrequencies of the EM spectrum. A photon detector may include an arrayof photon detectors, such as an arrangement of a plurality of photondetecting or sensing pixels. One or more of the pixels may be aphotosensor that is sensitive to the absorption of one or more photons.A photon detector may generate a signal in response to the absorption ofone or more photons. A photon detector may include a one-dimensional(1D) array of pixels. However, in other embodiments, photon detector mayinclude at least a two-dimensional (2D) array of pixels. The pixels mayinclude various photon-sensitive technologies, such as one or more ofactive-pixel sensors (APS), charge-coupled devices (CCDs), Single PhotonAvalanche Detector (SPAD) (operated in avalanche mode or Geiger mode),complementary metal-oxide-semiconductor (CMOS) devices, siliconphotomultipliers (SiPM), photovoltaic cells, phototransistors, twitchypixels, or the like. A photon detector may detect one or more incominglight beams.

As used herein, the term “target” is one or more various 2D or 3D bodiesthat reflect or scatter at least a portion of incident light, EM waves,or photons. The target may also be referred to as an “object.” Forinstance, a target or object may scatter or reflect an outgoing lightbeam that is transmitted by various ones of the various embodimentsdisclosed herein. In the various embodiments described herein, one ormore light sources may be in relative motion to one or more of receiversand/or one or more targets or objects. Similarly, one or more receiversmay be in relative motion to one or more of light sources and/or one ormore targets or objects. One or more targets or objects may be inrelative motion to one or more of light sources and/or one or morereceivers.

As used herein, the term “voxel” is a sampled surface element of a 3Dspatial manifold (for example, a 3D shaped surface.)

The following briefly describes embodiments of the invention in order toprovide a basic understanding of some aspects of the invention. Thisbrief description is not intended as an extensive overview. It is notintended to identify key or critical elements, or to delineate orotherwise narrow the scope. Its purpose is merely to present someconcepts in a simplified form as a prelude to the more detaileddescription that is presented later.

Briefly stated, various embodiments are directed to methods or systemsfor navigating a vehicle. A scanner is employed to scan a light beamover the surface. Light reflected by one or more fiducial markers on thesurface onto pixels of a receiver is employed to determine a spatialarrangement of the fiducial markers on the surface. The spatialarrangement of the fiducial markers is compared with a predetermined mapof the fiducial markers to determine a location of the vehicle.

Illustrated Operating Environment

FIG. 1 shows exemplary components of one embodiment of an exemplaryenvironment in which various exemplary embodiments of the invention maybe practiced. Not all of the components may be required to practice theinvention, and variations in the arrangement and type of the componentsmay be made without departing from the spirit or scope of the invention.As shown, system 100 of FIG. 1 includes a scanner 104 including a lightsource, at least one receiver (e.g., camera or sensor) 106, and a systemcomputer device 110. The scanner 104 has a light source that emits alight beam (e.g., photons) to sequentially illuminate regions (i.e.,voxels, points, lines, areas, or the like) on a surface 108, such as aroad, a landing strip, an autonomous vehicular lane or robotic workspace. The surface 108 includes fiducial markers 109 which reflect aportion of the light beam back towards the receiver 106. In someembodiments, system 100 may include, or be coupled to, a network 102 andone or more other computers, such as but not limited to a laptopcomputer 112 and/or a mobile computer, such as but not limited to asmartphone or tablet 114. In some embodiments, the scanner 104 and/orreceiver 106 may include one or more components included in a computer,such as but not limited to various ones of computers 110, 112, or 114.The scanner 104 and receiver 106 can be coupled directly to the computer110, 112, or 114 by any wireless or wired technique or may be coupled tothe computer 110, 112, or 114 through a network 102.

The scanner 104 may include one or more light sources for transmittinglight or photon beams. Examples of suitable light sources includeslasers, laser diodes, light emitting diodes, organic light emittingdiodes, or the like. For instance, the scanner 104 may include one ormore visible and/or non-visible laser sources. In at least someembodiments, the scanner 104 includes one or more of a white (W), red(R), a green (G), or a blue (B) light source. In at least someembodiments, the scanner 104 includes at least one each of a red (R), agreen (G), and a blue (B) light source. In at least some embodiment, thelight source includes one or more non-visible laser sources, such as anear-infrared (NIR) or infrared (IR) laser. A light source may providecontinuous or pulsed light beams of a predetermined frequency, or rangeof frequencies. The provided light beams may be coherent light beams.The scanner 104 may include various ones of the features, components, orfunctionality of a computer device, including but not limited to mobilecomputer 200 of FIG. 2 and/or network computer 300 of FIG. 3.

The scanner 104 may also include an optical system that includes opticalcomponents to direct or focus the transmitted or outgoing light beams.The optical systems may aim and shape the spatial and temporal beamprofiles of outgoing light beams. The optical system may collimate,fan-out, or otherwise manipulate the outgoing light beams. The scanner104 may include a scanning arrangement that can scan photons as a lightbeam over the surface 108. In at least some embodiments, the scanner 104may scan the light beam sequentially along a line or region (forexample, along voxels or points of the line or region) of the surfaceand then the scanner 104 may proceed to scan another line or region. Avoxel can be described as a sampled surface element of a 3D spatialmanifold (for example, the surface 108.) In at least some embodiments,the voxel is relatively small and may be described as “pixel-sized.” Insome embodiments, the scanner 105 may simultaneously illuminate a linewith the light beam and sequentially scan a series of lines on thesurface.

The receiver 106 may include one or more photon-sensitive, orphoton-detecting, arrays of sensor pixels. The terms “receiver”,“camera”, and “sensor” are used interchangeably herein and are used todenote any light or photon detector arrangement unless indicatedotherwise. An array of sensor pixels detects continuous or pulsed lightbeams reflected from the surface 108 or another target. The array ofpixels may be a one dimensional-array or a two-dimensional array. Thepixels may include SPAD pixels or other photo-sensitive elements thatavalanche upon the illumination one or a few incoming photons. Thepixels may have ultra-fast response times in detecting a single or a fewphotons that are on the order of a few nanoseconds. The pixels may besensitive to the frequencies emitted or transmitted by scanner 104 andrelatively insensitive to other frequencies. Receiver 106 can alsoinclude an optical system that includes optical components to direct andfocus the received beams, across the array of pixels. Receiver 106 mayinclude various ones of the features, components, or functionality of acomputer device, including but not limited to mobile computer 200 ofFIG. 2 and/or network computer 300 of FIG. 3.

Various embodiment of computer device 110 are described in more detailbelow in conjunction with FIGS. 2-3 (e.g., computer device 110 may be anembodiment of client or mobile computer 200 of FIG. 2 and/or networkcomputer 300 of FIG. 3). Briefly, however, computer device 110 includesvirtually various computer devices enabled to perform the variousnavigation or foveation operations, based on the detection of photonsreflected from one or more fiducial markers 109 in a surface. Based onthe detected photons or light beams, computer device 110 may alter orotherwise modify operation of a vehicle. It should be understood thatthe functionality of computer device 110 may be performed by scanner104, receiver 106, or a combination thereof, without communicating to aseparate device.

In some embodiments, at least some of the navigation or foveation orother functionality may be performed by other computers, including butnot limited to laptop computer 112 and/or a mobile computer, such as butnot limited to a smartphone or tablet 114. Various embodiments of suchcomputers are described in more detail below in conjunction with mobilecomputer 200 of FIG. 2 and/or network computer 300 of FIG. 3.

Network 102 may be configured to couple network computers with othercomputing devices, including scanner 104, photon receiver 106, trackingcomputer device 110, laptop computer 112, or smartphone/tablet 114.Network 102 may include various wired and/or wireless technologies forcommunicating with a remote device, such as, but not limited to, USBcable, Bluetooth®, or the like. In some embodiments, network 102 may bea network configured to couple network computers with other computingdevices. In various embodiments, information communicated betweendevices may include various kinds of information, including, but notlimited to, processor-readable instructions, remote requests, serverresponses, program modules, applications, raw data, control data, systeminformation (e.g., log files), video data, voice data, image data, textdata, structured/unstructured data, or the like. In some embodiments,this information may be communicated between devices using one or moretechnologies and/or network protocols.

In some embodiments, such a network may include various wired networks,wireless networks, or various combinations thereof. In variousembodiments, network 102 may be enabled to employ various forms ofcommunication technology, topology, computer-readable media, or thelike, for communicating information from one electronic device toanother. For example, network 102 can include—in addition to theInternet—LANs, WANs, Personal Area Networks (PANs), Campus AreaNetworks, Metropolitan Area Networks (MANs), direct communicationconnections (such as through a universal serial bus (USB) port), or thelike, or various combinations thereof.

In various embodiments, communication links within and/or betweennetworks may include, but are not limited to, twisted wire pair, opticalfibers, open air lasers, coaxial cable, plain old telephone service(POTS), wave guides, acoustics, full or fractional dedicated digitallines (such as T1, T2, T3, or T4), E-carriers, Integrated ServicesDigital Networks (ISDNs), Digital Subscriber Lines (DSLs), wirelesslinks (including satellite links), or other links and/or carriermechanisms known to those skilled in the art. Moreover, communicationlinks may further employ various ones of a variety of digital signalingtechnologies, including without limit, for example, DS-0, DS-1, DS-2,DS-3, DS-4, OC-3, OC-12, OC-48, or the like. In some embodiments, arouter (or other intermediate network device) may act as a link betweenvarious networks—including those based on different architectures and/orprotocols—to enable information to be transferred from one network toanother. In other embodiments, remote computers and/or other relatedelectronic devices could be connected to a network via a modem andtemporary telephone link. In essence, network 102 may include variouscommunication technologies by which information may travel betweencomputing devices.

Network 102 may, in some embodiments, include various wireless networks,which may be configured to couple various portable network devices,remote computers, wired networks, other wireless networks, or the like.Wireless networks may include various ones of a variety of sub-networksthat may further overlay stand-alone ad-hoc networks, or the like, toprovide an infrastructure-oriented connection for at least clientcomputer (e.g., laptop computer 112 or smart phone or tablet computer114) (or other mobile devices). Such sub-networks may include meshnetworks, Wireless LAN (WLAN) networks, cellular networks, or the like.In one or more of the various embodiments, the system may include morethan one wireless network.

Network 102 may employ a plurality of wired and/or wirelesscommunication protocols and/or technologies. Examples of variousgenerations (e.g., third (3G), fourth (4G), or fifth (5G)) ofcommunication protocols and/or technologies that may be employed by thenetwork may include, but are not limited to, Global System for Mobilecommunication (GSM), General Packet Radio Services (GPRS), Enhanced DataGSM Environment (EDGE), Code Division Multiple Access (CDMA), WidebandCode Division Multiple Access (W-CDMA), Code Division Multiple Access2000 (CDMA2000), High Speed Downlink Packet Access (HSDPA), Long TermEvolution (LTE), Universal Mobile Telecommunications System (UMTS),Evolution-Data Optimized (Ev-DO), Worldwide Interoperability forMicrowave Access (WiMax), time division multiple access (TDMA),Orthogonal frequency-division multiplexing (OFDM), ultra-wide band(UWB), Wireless Application Protocol (WAP), user datagram protocol(UDP), transmission control protocol/Internet protocol (TCP/IP), variousportions of the Open Systems Interconnection (OSI) model protocols,session initiated protocol/real-time transport protocol (SIP/RTP), shortmessage service (SMS), multimedia messaging service (MMS), or variousones of a variety of other communication protocols and/or technologies.In essence, the network may include communication technologies by whichinformation may travel between scanner 104, photon receiver 106, andtracking computer device 110, as well as other computing devices notillustrated.

In various embodiments, at least a portion of network 102 may bearranged as an autonomous system of nodes, links, paths, terminals,gateways, routers, switches, firewalls, load balancers, forwarders,repeaters, optical-electrical converters, or the like, which may beconnected by various communication links. These autonomous systems maybe configured to self-organize based on current operating conditionsand/or rule-based policies, such that the network topology of thenetwork may be modified.

Illustrative Client Computer

FIG. 2 shows one embodiment of an exemplary client computer 200 that mayinclude many more or less components than those exemplary componentsshown. Client computer 200 may represent, for example, one or moreembodiment of laptop computer 112, smartphone/tablet 114, and/orcomputer 110 of system 100 of FIG. 1. Thus, client computer 200 mayinclude a mobile device (e.g., a smart phone or tablet), astationary/desktop computer, or the like.

Client computer 200 may include processor 202 in communication withmemory 204 via bus 206. Client computer 200 may also include powersupply 208, network interface 210, processor-readable stationary storagedevice 212, processor-readable removable storage device 214,input/output interface 216, camera(s) 218, video interface 220, touchinterface 222, hardware security module (HSM) 224, projector 226,display 228, keypad 230, illuminator 232, audio interface 234, globalpositioning systems (GPS) transceiver 236, open air gesture interface238, temperature interface 240, haptic interface 242, and pointingdevice interface 244. Client computer 200 may optionally communicatewith a base station (not shown), or directly with another computer. Andin one embodiment, although not shown, a gyroscope may be employedwithin client computer 200 for measuring and/or maintaining anorientation of client computer 200.

Power supply 208 may provide power to client computer 200. Arechargeable or non-rechargeable battery may be used to provide power.The power may also be provided by an external power source, such as anAC adapter or a powered docking cradle that supplements and/or rechargesthe battery.

Network interface 210 includes circuitry for coupling client computer200 to one or more networks, and is constructed for use with one or morecommunication protocols and technologies including, but not limited to,protocols and technologies that implement various portions of the OSImodel for mobile communication (GSM), CDMA, time division multipleaccess (TDMA), UDP, TCP/IP, SMS, MMS, GPRS, WAP, UWB, WiMax, SIP/RTP,GPRS, EDGE, WCDMA, LTE, UMTS, OFDM, CDMA2000, EV-DO, HSDPA, or variousones of a variety of other wireless communication protocols. Networkinterface 210 is sometimes known as a transceiver, transceiving device,or network interface card (NIC).

Audio interface 234 may be arranged to produce and receive audio signalssuch as the sound of a human voice. For example, audio interface 234 maybe coupled to a speaker and microphone (not shown) to enabletelecommunication with others and/or generate an audio acknowledgementfor some action. A microphone in audio interface 234 can also be usedfor input to or control of client computer 200, e.g., using voicerecognition, detecting touch based on sound, and the like.

Display 228 may be a liquid crystal display (LCD), gas plasma,electronic ink, light emitting diode (LED), Organic LED (OLED) orvarious other types of light reflective or light transmissive displaysthat can be used with a computer. Display 228 may also include the touchinterface 222 arranged to receive input from an object such as a stylusor a digit from a human hand, and may use resistive, capacitive, surfaceacoustic wave (SAW), infrared, radar, or other technologies to sensetouch and/or gestures.

Projector 226 may be a remote handheld projector or an integratedprojector that is capable of projecting an image on a remote wall orvarious other reflective objects such as a remote screen.

Video interface 220 may be arranged to capture video images, such as astill photo, a video segment, an infrared video, or the like. Forexample, video interface 220 may be coupled to a digital video camera, aweb-camera, or the like. Video interface 220 may comprise a lens, animage sensor, and other electronics. Image sensors may include acomplementary metal-oxide-semiconductor (CMOS) integrated circuit,charge-coupled device (CCD), or various other integrated circuits forsensing light.

Keypad 230 may comprise various input devices arranged to receive inputfrom a user. For example, keypad 230 may include a push button numericdial, or a keyboard. Keypad 230 may also include command buttons thatare associated with selecting and sending images.

Illuminator 232 may provide a status indication and/or provide light.Illuminator 232 may remain active for specific periods of time or inresponse to event messages. For example, if illuminator 232 is active,it may backlight the buttons on keypad 230 and stay on while the clientcomputer is powered. Also, illuminator 232 may backlight these buttonsin various patterns if particular actions are performed, such as dialinganother client computer. Illuminator 232 may also cause light sourcespositioned within a transparent or translucent case of the clientcomputer to illuminate in response to actions.

Further, client computer 200 may also comprise HSM 224 for providingadditional tamper resistant safeguards for generating, storing and/orusing security/cryptographic information such as, keys, digitalcertificates, passwords, passphrases, two-factor authenticationinformation, or the like. In some embodiments, hardware security modulemay be employed to support one or more standard public keyinfrastructures (PKI), and may be employed to generate, manage, and/orstore keys pairs, or the like. In some embodiments, HSM 224 may be astand-alone computer, in other cases, HSM 224 may be arranged as ahardware card that may be added to a client computer.

Client computer 200 may also comprise input/output interface 216 forcommunicating with external peripheral devices or other computers suchas other client computers and network computers. The peripheral devicesmay include an audio headset, virtual reality headsets, display screenglasses, remote speaker system, remote speaker and microphone system,and the like. Input/output interface 216 can utilize one or moretechnologies, such as Universal Serial Bus (USB), Infrared, Wi-Fi™,WiMax, Bluetooth™, and the like.

Input/output interface 216 may also include one or more sensors fordetermining geolocation information (e.g., GPS), monitoring electricalpower conditions (e.g., voltage sensors, current sensors, frequencysensors, and so on), monitoring weather (e.g., thermostats, barometers,anemometers, humidity detectors, precipitation scales, or the like), orthe like. Sensors may be one or more hardware sensors that collectand/or measure data that is external to client computer 200.

Haptic interface 242 may be arranged to provide tactile feedback to auser of the client computer. For example, the haptic interface 242 maybe employed to vibrate client computer 200 in a particular way ifanother user of a computer is calling. Temperature interface 240 may beused to provide a temperature measurement input and/or a temperaturechanging output to a user of client computer 200. Open air gestureinterface 238 may sense physical gestures of a user of client computer200, for example, by using single or stereo video cameras, radar, agyroscopic sensor inside a computer held or worn by the user, or thelike. Camera 218 may be used to track physical eye movements of a userof client computer 200.

GPS transceiver 236 can determine the physical coordinates of clientcomputer 200 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 236 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference(E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), EnhancedTiming Advance (ETA), Base Station Subsystem (BSS), or the like, tofurther determine the physical location of client computer 200 on thesurface of the Earth. It is understood that under different conditions,GPS transceiver 236 can determine a physical location for clientcomputer 200. In one or more embodiments, however, client computer 200may, through other components, provide other information that may beemployed to determine a physical location of the client computer,including for example, a Media Access Control (MAC) address, IP address,and the like.

Human interface components can be peripheral devices that are physicallyseparate from client computer 200, allowing for remote input and/oroutput to client computer 200. For example, information routed asdescribed here through human interface components such as display 228 orkeypad 230 can instead be routed through network interface 210 toappropriate human interface components located remotely. Examples ofhuman interface peripheral components that may be remote include, butare not limited to, audio devices, pointing devices, keypads, displays,cameras, projectors, and the like. These peripheral components maycommunicate over a Pico Network such as Bluetooth™, Zigbee™ and thelike. One non-limiting example of a client computer with such peripheralhuman interface components is a wearable computer, which might include aremote pico projector along with one or more cameras that remotelycommunicate with a separately located client computer to sense a user'sgestures toward portions of an image projected by the pico projectoronto a reflected surface such as a wall or the user's hand.

Memory 204 may include RAM, ROM, and/or other types of memory. Memory204 illustrates an example of computer-readable storage media (devices)for storage of information such as computer-readable instructions, datastructures, program modules or other data. Memory 204 may store BIOS 246for controlling low-level operation of client computer 200. The memorymay also store operating system 248 for controlling the operation ofclient computer 200. It will be appreciated that this component mayinclude a general-purpose operating system such as a version of UNIX, orLINUX™, or a specialized client computer communication operating systemsuch as Windows Phone™, or the Symbian® operating system. The operatingsystem may include, or interface with a Java virtual machine module thatenables control of hardware components and/or operating systemoperations via Java application programs.

Memory 204 may further include one or more data storage 250, which canbe utilized by client computer 200 to store, among other things,applications 252 and/or other data. For example, data storage 250 mayalso be employed to store information that describes variouscapabilities of client computer 200. In one or more of the variousembodiments, data storage 250 may store a fiducial marker map or roadsurface map 251. The map 251 may then be provided to another device orcomputer based on various ones of a variety of methods, including beingsent as part of a header during a communication, sent upon request, orthe like. Data storage 250 may also be employed to store socialnetworking information including address books, buddy lists, aliases,user profile information, or the like. Data storage 250 may furtherinclude program code, data, algorithms, and the like, for use by aprocessor, such as processor 202 to execute and perform actions. In oneembodiment, at least some of data storage 250 might also be stored onanother component of client computer 200, including, but not limited to,non-transitory processor-readable stationary storage device 212,processor-readable removable storage device 214, or even external to theclient computer.

Applications 252 may include computer executable instructions which, ifexecuted by client computer 200, transmit, receive, and/or otherwiseprocess instructions and data. Applications 252 may include, forexample, navigation client engine 253, foveation client engine 254,other client engines 256, web browser 258, or the like. Client computersmay be arranged to exchange communications, such as, queries, searches,messages, notification messages, event messages, alerts, performancemetrics, log data, API calls, or the like, combination thereof, withapplication servers, network file system applications, and/or storagemanagement applications.

The web browser engine 226 may be configured to receive and to send webpages, web-based messages, graphics, text, multimedia, and the like. Theclient computer's browser engine 226 may employ virtually variousprogramming languages, including a wireless application protocolmessages (WAP), and the like. In one or more embodiments, the browserengine 258 is enabled to employ Handheld Device Markup Language (HDML),Wireless Markup Language (WML), WMLScript, JavaScript, StandardGeneralized Markup Language (SGML), HyperText Markup Language (HTML),eXtensible Markup Language (XML), HTMLS, and the like.

Other examples of application programs include calendars, searchprograms, email client applications, IM applications, SMS applications,Voice Over Internet Protocol (VOIP) applications, contact managers, taskmanagers, transcoders, database programs, word processing programs,security applications, spreadsheet programs, games, search programs, andso forth.

Additionally, in one or more embodiments (not shown in the figures),client computer 200 may include an embedded logic hardware deviceinstead of a CPU, such as, an Application Specific Integrated Circuit(ASIC), Field Programmable Gate Array (FPGA), Programmable Array Logic(PAL), or the like, or combination thereof. The embedded logic hardwaredevice may directly execute its embedded logic to perform actions. Also,in one or more embodiments (not shown in the figures), client computer200 may include a hardware microcontroller instead of a CPU. In one ormore embodiments, the microcontroller may directly execute its ownembedded logic to perform actions and access its own internal memory andits own external Input and Output Interfaces (e.g., hardware pins and/orwireless transceivers) to perform actions, such as System On a Chip(SOC), or the like.

Illustrative Network Computer

FIG. 3 shows one embodiment of an exemplary network computer 300 thatmay be included in an exemplary system implementing one or more of thevarious embodiments. Network computer 300 may include many more or lesscomponents than those shown in FIG. 3. However, the components shown aresufficient to disclose an illustrative embodiment for practicing theseinnovations. Network computer 300 may include a desktop computer, alaptop computer, a server computer, a client computer, and the like.Network computer 300 may represent, for example, one embodiment of oneor more of laptop computer 112, smartphone/tablet 114, and/or computer110 of system 100 of FIG. 1.

As shown in FIG. 3, network computer 300 includes a processor 302 thatmay be in communication with a memory 304 via a bus 306. In someembodiments, processor 302 may be comprised of one or more hardwareprocessors, or one or more processor cores. In some cases, one or moreof the one or more processors may be specialized processors designed toperform one or more specialized actions, such as, those describedherein. Network computer 300 also includes a power supply 308, networkinterface 310, processor-readable stationary storage device 312,processor-readable removable storage device 314, input/output interface316, GPS transceiver 318, display 320, keyboard 322, audio interface324, pointing device interface 326, and HSM 328. Power supply 308provides power to network computer 300.

Network interface 310 includes circuitry for coupling network computer300 to one or more networks, and is constructed for use with one or morecommunication protocols and technologies including, but not limited to,protocols and technologies that implement various portions of the OpenSystems Interconnection model (OSI model), global system for mobilecommunication (GSM), code division multiple access (CDMA), time divisionmultiple access (TDMA), user datagram protocol (UDP), transmissioncontrol protocol/Internet protocol (TCP/IP), Short Message Service(SMS), Multimedia Messaging Service (MMS), general packet radio service(GPRS), WAP, ultra wide band (UWB), IEEE 802.16 WorldwideInteroperability for Microwave Access (WiMax), Session InitiationProtocol/Real-time Transport Protocol (SIP/RTP), or various ones of avariety of other wired and wireless communication protocols. Networkinterface 310 is sometimes known as a transceiver, transceiving device,or network interface card (NIC). Network computer 300 may optionallycommunicate with a base station (not shown), or directly with anothercomputer.

Audio interface 324 is arranged to produce and receive audio signalssuch as the sound of a human voice. For example, audio interface 324 maybe coupled to a speaker and microphone (not shown) to enabletelecommunication with others and/or generate an audio acknowledgementfor some action. A microphone in audio interface 324 can also be usedfor input to or control of network computer 300, for example, usingvoice recognition.

Display 320 may be a liquid crystal display (LCD), gas plasma,electronic ink, light emitting diode (LED), Organic LED (OLED) orvarious other types of light reflective or light transmissive displaythat can be used with a computer. Display 320 may be a handheldprojector or pico projector capable of projecting an image on a wall orother object.

Network computer 300 may also comprise input/output interface 316 forcommunicating with external devices or computers not shown in FIG. 3.Input/output interface 316 can utilize one or more wired or wirelesscommunication technologies, such as USB™, Firewire™, Wi-Fi™ WiMax,Thunderbolt™, Infrared, Bluetooth™, Zigbee™, serial port, parallel port,and the like.

Also, input/output interface 316 may also include one or more sensorsfor determining geolocation information (e.g., GPS), monitoringelectrical power conditions (e.g., voltage sensors, current sensors,frequency sensors, and so on), monitoring weather (e.g., thermostats,barometers, anemometers, humidity detectors, precipitation scales, orthe like), or the like. Sensors may be one or more hardware sensors thatcollect and/or measure data that is external to network computer 300.Human interface components can be physically separate from networkcomputer 300, allowing for remote input and/or output to networkcomputer 300. For example, information routed as described here throughhuman interface components such as display 320 or keyboard 322 caninstead be routed through the network interface 310 to appropriate humaninterface components located elsewhere on the network. Human interfacecomponents include various components that allow the computer to takeinput from, or send output to, a human user of a computer. Accordingly,pointing devices such as mice, styluses, track balls, or the like, maycommunicate through pointing device interface 326 to receive user input.

GPS transceiver 318 can determine the physical coordinates of networkcomputer 300 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 318 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference(E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), EnhancedTiming Advance (ETA), Base Station Subsystem (BSS), or the like, tofurther determine the physical location of network computer 300 on thesurface of the Earth. It is understood that under different conditions,GPS transceiver 318 can determine a physical location for networkcomputer 300. In one or more embodiments, however, network computer 300may, through other components, provide other information that may beemployed to determine a physical location of the client computer,including for example, a Media Access Control (MAC) address, IP address,and the like.

Memory 304 may include Random Access Memory (RAM), Read-Only Memory(ROM), and/or other types of memory. Memory 304 illustrates an exampleof computer-readable storage media (devices) for storage of informationsuch as computer-readable instructions, data structures, program modulesor other data. Memory 304 stores a basic input/output system (BIOS) 330for controlling low-level operation of network computer 300. The memoryalso stores an operating system 332 for controlling the operation ofnetwork computer 300. It will be appreciated that this component mayinclude a general-purpose operating system such as a version of UNIX, orLINUX™, or a specialized operating system such as MicrosoftCorporation's Windows® operating system, or the Apple Corporation's IOS®operating system. The operating system may include, or interface with aJava virtual machine module that enables control of hardware componentsand/or operating system operations via Java application programs.Likewise, other runtime environments may be included.

Memory 304 may further include one or more data storage 334, which canbe utilized by network computer 300 to store, among other things,applications 336 and/or other data. For example, data storage 334 mayalso be employed to store information that describes variouscapabilities of network computer 300. In one or more of the variousembodiments, data storage 334 may store a fiducial marker map or a roadsurface map 335. The fiducial marker map or a road surface map 335 maythen be provided to another device or computer based on various ones ofa variety of methods, including being sent as part of a header during acommunication, sent upon request, or the like. Data storage 334 may alsobe employed to store social networking information including addressbooks, buddy lists, aliases, user profile information, or the like. Datastorage 334 may further include program code, data, algorithms, and thelike, for use by one or more processors, such as processor 302 toexecute and perform actions such as those actions described below. Inone embodiment, at least some of data storage 334 might also be storedon another component of network computer 300, including, but not limitedto, non-transitory media inside non-transitory processor-readablestationary storage device 312, processor-readable removable storagedevice 314, or various other computer-readable storage devices withinnetwork computer 300, or even external to network computer 300.

Applications 336 may include computer executable instructions which, ifexecuted by network computer 300, transmit, receive, and/or otherwiseprocess messages (e.g., SMS, Multimedia Messaging Service (MMS), InstantMessage (IM), email, and/or other messages), audio, video, and enabletelecommunication with another user of another mobile computer. Otherexamples of application programs include calendars, search programs,email client applications, IM applications, SMS applications, Voice OverInternet Protocol (VOIP) applications, contact managers, task managers,transcoders, database programs, word processing programs, securityapplications, spreadsheet programs, games, search programs, and soforth. Applications 336 may include navigation engine 344 or foveationengine 346 that performs actions further described below. In one or moreof the various embodiments, one or more of the applications may beimplemented as modules and/or components of another application.Further, in one or more of the various embodiments, applications may beimplemented as operating system extensions, modules, plugins, or thelike.

Furthermore, in one or more of the various embodiments, navigationengine 344 or foveation engine 346 may be operative in a cloud-basedcomputing environment. In one or more of the various embodiments, theseapplications, and others, may be executing within virtual machinesand/or virtual servers that may be managed in a cloud-based basedcomputing environment. In one or more of the various embodiments, inthis context the applications may flow from one physical networkcomputer within the cloud-based environment to another depending onperformance and scaling considerations automatically managed by thecloud computing environment. Likewise, in one or more of the variousembodiments, virtual machines and/or virtual servers dedicated tonavigation engine 344 or foveation engine 346 may be provisioned andde-commissioned automatically.

Also, in one or more of the various embodiments, navigation engine 344or foveation engine 346 or the like may be located in virtual serversrunning in a cloud-based computing environment rather than being tied toone or more specific physical network computers.

Further, network computer 300 may comprise HSM 328 for providingadditional tamper resistant safeguards for generating, storing and/orusing security/cryptographic information such as, keys, digitalcertificates, passwords, passphrases, two-factor authenticationinformation, or the like. In some embodiments, hardware security modulemay be employed to support one or more standard public keyinfrastructures (PKI), and may be employed to generate, manage, and/orstore keys pairs, or the like. In some embodiments, HSM 328 may be astand-alone network computer, in other cases, HSM 328 may be arranged asa hardware card that may be installed in a network computer.

Additionally, in one or more embodiments (not shown in the figures), thenetwork computer may include one or more embedded logic hardware devicesinstead of one or more CPUs, such as, an Application Specific IntegratedCircuits (ASICs), Field Programmable Gate Arrays (FPGAs), ProgrammableArray Logics (PALs), or the like, or combination thereof. The embeddedlogic hardware devices may directly execute embedded logic to performactions. Also, in one or more embodiments (not shown in the figures),the network computer may include one or more hardware microcontrollersinstead of a CPU. In one or more embodiments, the one or moremicrocontrollers may directly execute their own embedded logic toperform actions and access their own internal memory and their ownexternal Input and Output Interfaces (e.g., hardware pins and/orwireless transceivers) to perform actions, such as System On a Chip(SOC), or the like.

Illustrative Systems

In complex motion systems it is advantageous that various observers havean accurate, common spatial-temporal reference (for example, a shared“ground truth,” or a “terra firma” to stand on.) In the laboratory thepurpose of the optical bench is to eliminate the vibrations of theexternal world. In the laboratory not only motion but also time ishighly controlled. Computer vision and other high-performance computingsystems often use nanosecond precise clock references.

However, in contrast, the real world is noisy and chaotic. Everything isin constant motion. Signal propagation takes significant time and isoften variable, causing signal jitter (for example, unpredictablevarying time delays). Furthermore, digital sensors inherently generatequantization errors; data buffers introduce latency; and real worldevents appear to occur randomly in time, not synchronized to a precisemaster clock.

High-speed motion tracking systems currently in development (forexample, for robotic vision, autonomous navigation, and numerous otherapplications) have created a need for more precise measurement ofspatial dimensions and their time functions. The over-arching objectiveis to accurately detect, identify and track objects, estimate theirpositions and motions, and arrive at reliable predictions of theirtrajectories and deformations.

Most, if not all sensors (including sensors for detecting light orphotons) observe relative position and motion, relative velocity (e.g.Doppler acoustics or radar), or change of motion (e.g. inertial sensors)and it is often desirable or even crucial in tracking or positionalnavigational systems to have some kind of absolute reference or groundtruth position on which to anchor a shared coordinate system. Withoutsuch an anchor achieving sufficient accuracy can be a challenge,especially in high-velocity, dynamic, or chaotic systems where rapidchanges occur in an un-anticipated and unpredictable manner, and wherethe sensors themselves are in a constant (often unknown or poorlydefined) state of motion.

The real world does provide a natural anchoring system: it is the groundwe stand on, “Terra Firma” (Latin: solid ground) and which can serve asa global map of the environment, onto which knowledge from all sensors(including sensors for detecting light or photons) can accumulate. Forprecise navigation such a highly detailed global map is as useful asancient maps of “Terra Cognita” (Latin: the Known World) were fornavigators such Magellan and Columbus. The greater the accuracy, thegreater the value and the more information can be synergisticallyaccumulated through successive observations. For example, highlyaccurate 3D observations from successive independent observers (e.g., byLIDAR or other observation techniques) can enable a crowd sourced evermore fine-grained view of cities, their drivable surfaces, andobservable structures.

Light can be used as a spatio-temporal reference. Some recentlydeveloped systems, such as for example, scanning LIDARs, use thesequential illumination of the 3D space. A unique, highly accurate 3Dmotion capture system architecture is the PhotonJet™ imagingarchitecture, versions of which have been described in U.S. Pat. Nos.8,282,888; 8,430,512; 8,573,783; 8,696,141; 8,711,370; 8,971,568;9,377,533; 9,501,176; 9,581,883; 9,753,126; 9,810,913; 9,813,673;9,946,076; 10,043,282; 10,061,137; 10,067,230; and Ser. No. 10/084,990;and U.S. patent application Ser. Nos. 15/853,783 and 15/976,269, all ofwhich are incorporated herein by reference in their entirety. In a pixelsequential scanning architecture (such as the PhotonJet′ imagingarchitecture), individual pixels (spatial light contrast measurements)and voxels (3D surface position measurements) are observed with pinpointprecision. For example, in at least some embodiments of the PhotonJet™imaging architecture, the pixels can be measured with a precision of a1/100^(th) of degree and resolved in time in nanosecond intervals.

Cameras are everywhere. In the last 10 years CMOS camera technology hasadvanced dramatically, in resolution and quantum efficiency, and at thesame time their cost has dropped dramatically, a trend primarily drivenby mass consumer applications. A 10 megapixel color camera module maynow cost less than $10. The same technology curve will in principleenable a 100-megapixel monochrome position sensor IC.

Mass produced, low cost optics developed for cell phone cameras enablethe affordable sensor arrays (arrays of low cost camera modules) thatare found in 360 degree surround cameras used for VR capture and mountedonto quad copters and autonomous cars, test driving on the roads in theSan Francisco Bay Area, Marseille and Singapore. Such “robottransporters” are often festooned with arrays of 10 or more individual,high-resolution sensors.

Properly deployed, such systems are capable of capturing an extremelyfine-grained view of the real world around them. For example, a highresolution “4K” camera has 8,000,000 pixels typically arranged in 4000columns (hence “4 k”) and 2000 rows. As an example, with good optics, alens stack with 40 by 20 degree field of view, observes, resolves,measures and tracks the world to 100^(th) of a degree, resolving oneinch details at 477 feet (1 cm details at 57.3 m). Tele-focus systems,(such as some satellite and UAV vision systems) with high qualitytelescopic lenses observe the finest details and are limited by the costof the system and the laws of optics.

Motion, however, is the enemy of resolution. As photographers have knownsince the invention of photography 180 years ago, to create a nicefamily portrait you need good light and a tripod, and everyone has tosit still. High speed object observation takes specially arranged lightsources e.g. the high-speed photography using mechanically triggeredcamera arrays pioneered by Muybridge in 1878 at the Stanford ranch tocapture the gait of a galloping horse.

High altitude orbital satellites can see that amazing detail in yourbackyard because they drift silently, weightlessly through space in aperfectly balanced equilibrium of kinetic and gravitational forces. Onlythe laws of optics and atmospheric factors such as the weather limittheir resolution. An ideal combination of large optics, perfectly cooledsensors and a total absence of vibrations enable leisurely longexposures that result in extremely accurate, noise free, giga-pixelimages from high above in our sky.

On or just above the earth's surface, achieving even a 1000^(th) of thatimage quality is challenging. Unfortunately for UAVs, quad copters,Paparazzi on motorcycles and (soon) autonomous taxis, even the besthigh-speed sensors and state of the art motion stabilizers cannot fullyeliminate the debilitating effects of a bumpy ride, the vibration ofmotors, rotors and a shaking fuselage. Action equals reaction: the samefundamentals from Newtonian physics that enable the supreme quality ofsatellite images impose drastic limits on the performance and imagequality of non-celestial systems.

Accurate high-velocity observation is challenging when high-speedmotions are observed and experienced at the same time. High pixel countsenable accuracy, but to achieve sharp image contrast (for example, toobserve an edge of a vehicle racing towards the finish line) each pixelmay need at least 1000 photons. The higher the pixel count and thegreater the distance the more that becomes a problem. The problem isinsufficiency of instantaneously available photons (“photonstarvation”.) Given a certain lens aperture size even the best qualityoptics can only capture an infinitesimally small proportion of the lightreflected off or emitted by a far object.

Global shutter cameras are often used in high-speed motion capture andcurrently dominate autonomous navigation applications. In such systemsonly one photon may on average arrive at any one of the pixels in thecamera in any microsecond, so an exposure time of at least 1 millisecondmay be required to achieve a sufficiently sharp photo-finish image. At ashort 1-millisecond exposure a speeding car (50 m/s) may still move 2inches or more. A high-resolution camera with, for example, 40 degreeslateral view and a 4K sensor has 100 pixels per degree across the fieldof view. Such a 4K camera observing a vehicle crossing the finish linefrom a 5-meter distance would experience a significant motion blur ofmore than 50 pixels. However, to reduce this unfortunate motion blur toonly one pixel the light intensity would be increased 50× duringexposure to reduce the exposure time to 1/50^(th), that is, to 20microseconds rather than 1 millisecond.

Camera motion means short frame exposure times to reduce blur. Anothersignificant challenge is that all motorized transport platforms vibratedue to motor vibrations and wheel and/or rotor impacts. As an example,at 6000 RMP a Tesla model 3 engine generates mechanical vibrations at100 Hz while providing 100 kW of power enabling it to accelerate from 0to 60 mph in 5.6 seconds. The motor's AC drive system creates additionalhigher frequency harmonic vibrations.

To improve the power-to-weight ratio, various electric vehicle (EV)suppliers are working on powerful but light EV motors that will run at20,000 RPM. These vehicles generate mechanical shock waves atfrequencies between 100 to 333 Hz. All mechanical things vibrate andthese low frequency vibrations will propagate despite the best dampeningtechniques. The frequencies of these vibrations are similar to thepowerful bass sound frequencies generated by car audio systems. Lowfrequencies are very energetic and fundamentally hard to absorb. Suchvibrations inevitably reach the cameras and other sensors in autonomousmobility platforms. There is an ongoing industry effort to deliver aquieter, smoother ride by, for example, deploying noise-canceling shockabsorbers and cabin active noise cancellation systems, such as pioneeredby Bose™ acoustics.

However, any transport platform (i.e., vehicle) when moving at greatspeed through air will experience some turbulence. Navigational camerasare typically embedded in the front surface of the hull against whichair collides more turbulently as the vehicle's velocity increases.

To navigate at greater speed autonomous vehicles should see traffic infine detail further ahead. At 70 mph or higher, 4K resolution (1/100^(th) degree voxels) may be needed. In at least some instances,this is the resolution to be able to distinguish at a sufficientdistance a pedestrian from a cyclist or a car sharing a road surface.Such resolution insures that there is sufficient time to detect anobstacle, to make the appropriate classification, and to choose andsuccessfully execute the correct collision avoidance maneuver.

FIG. 4 is a diagram illustrating an example of camera vibration. Assumea camera module vibrates at 100 Hz+/−1 degree diagonally (as illustratedby line 490) across the field of view, as illustrated in FIG. 1. Thiswill move the camera at a speed of up to 400 degrees per second (i.e.,two degrees in 5 ms). If, at rest, the camera's optics and sensor canresolve images to 1/100th of a degree, then a 100 Hz vibration may causerotational vibration blur (blur caused by the camera's optical axisrotating) of 40,000 pixels per second (400 degrees/s multiplied by 100pixels per degree). From the camera's perspective the world is shakingacross its field of view with fine detail flying from one pixel to thenext adjacent pixel every 25 microseconds ( 1/40,000^(th) of a second).This implies that nothing close to the optical specification of thecamera system can be observed once the car starts accelerating. Forexample, when an autonomous car merges from an on-ramp and acceleratesonto a freeway, it will need to focus its vision system farther ahead tocheck the road for obstructions and plan the correct merge maneuver, butthis blurring can be a challenge to that vision system. The samecritical requirements exist when passing or avoiding a sudden roadhazard on an undivided two-lane country road. Oncoming traffic needs tobe spotted and identified. The velocity heading and likely trajectoryneeds to be estimated correctly with short latency at significantdistances. With at best only a few seconds to a frontal collision, anycamera motion blur could reduce the system's perception capabilities andability to avoid collision.

Illumination can be key factor. Headlights are less powerful at greaterdistances due to 1/R² reflection losses. New automotive lighting systemsaim for greater and more concentrated power, employing high performanceLED arrays collimated into highly focused but non-blinding “smart” highbeams. Since longer exposure times are often not an option to make morephotons available, greater camera apertures may use expensive full framesize sensors (24 by 36 mm) coupled with expensive optics.

Current automotive vision systems tend to employ global shutter cameraswith an enhanced dynamic range. Additional pixel logic may be needed tosupport high frame-rates feeding pixel data into accurate analog todigital converters, driving up the cost of these systems very rapidly.

In contrast to conventional arrangements, in at least some embodiments,a sequential image acquisition system, such a that illustrated in FIG.1, can take just a few nanoseconds to acquire a precise voxel positionestimation when scanning a 3D manifold using methods that are describedin the various PhotonJet′ imaging architecture patent documentsreferenced above. In a pixel sequential image acquisition systems (also,called pixel sequential perception systems) a laser beam scanssuccessive pixel-sized segments in the field of view (FoV). The short,intense laser illumination of individual voxels reduces or eliminatesmotion blur, because there can be no significant shift in positionbetween the camera and the voxel during acquisition. During a10-nanosecond pixel illumination, 10⁻⁸ seconds, a speeding car with avelocity of 50 m/s (180 km/h) moves only half a micron (50 m/s for 10nanoseconds). In line sequential image acquisition systems (also calledline sequential perception systems) the light source can simultaneouslyilluminate any entire line (or a multi-pixel portion of a line). Pixeland line sequential image acquisition systems both include the lightsource 106 and camera 104 illustrated in FIG. 1.

A pixel sequential image acquisition system (which can include thePhotonJet™ imaging architecture described in the references citedherein) may complete the scan of an entire line of 2000 pixels (forexample, 2000 individual sequential observations of approximately onesquare millimeter of road surface) in approximately 20 microseconds(2×10⁻⁵ sec).

FIG. 5A illustrates a vehicle 550 with a pixel sequential perceptionsystem that sequentially scans a line 554 of a surface 552 (such as aroad) divided into 2000 positions 556 (e.g., pixels or voxels) at a rateof 10 ns/position to provide a total scan time of the line 554 at 20microseconds. Alternatively, in at least some embodiments, a linesequential perception system, such as a system with laser strobeheadlights, can illuminate the whole line 554 simultaneously for 20microseconds. In that time interval a car speeding at 50 meters persecond moves only one millimeter.

FIG. 5B illustrates a line sequential perception system 500 that mayinclude a light source 506, such as a laser, (and any associated optics511) that illuminates a line 554 (or in other embodiments of pixelsequential perception systems, sequentially illuminates pixels,positions, or voxels along a line) and is synchronized with a linearpixel array camera 504 or other sensor or receiver (and any associatedoptics such as a lens 515). As an example, the 445 nm light from apowerful (for example, 3 W) blue laser diode can be collimated along itsfast axis into a sharply focused laser line illumination of the roadsurface. Such a system rakes a narrow stripe across the field of view,such as the road just ahead of the vehicle. In at least someembodiments, the light source 506 and camera 504 are located in one orboth headlights of the vehicle 550. In other embodiments, these elementsmay be located elsewhere on a vehicle.

Alternatively, a line sequential perception system may produce a scanline that moves vertically across a rectangular FoV strobing successivelines along the road. If each line takes 20 microseconds, a system cantake in 50,000 lines per second. If the focus was set to illuminate andview a region that is 1 mm in width per line, then the system couldcover the entire drivable road surface ahead, each square mm, at speedsup to 180 km per hour. At slower speeds the successive frames acquiredcould overlap. For example, at a more moderate driving speed of 25meters per second (approx. 55 mph) the successive frames could overlapby 50%.

In this manner, illumination, either in a pixel line sequentialperception system or a line sequential perception system, can reduce oreliminate sources of motion blur. FIG. 6 illustrates one embodiment of asystem that includes one or more light sources (not shown), for example,lasers or light-emitting diodes, that scan a light beam across the fieldof view 631. One or more cameras 604 observe the reflections of thelight beam as it illuminates a surface 652, such as a road surface.Small, preferably retro-reflective, fiducial markers 609 are embedded inthe surface in an irregular pattern. These fiducial markers 609 stronglyand preferentially reflect the scanning beam's light back to thecamera(s) 604. These point reflections are focused by the camera opticsinto sharp illuminated spots 637 on the camera's surface. One such spot637 may be approximately the size of a pixel in the camera 604, or thespot may be smaller than the pixel, falling wholly within a pixel attimes, or it may be larger than a pixel and illuminate several pixels atthe same time. In the latter case a centroid of the spot may becalculated by a processor using, for example, the greyscale of each ofthe pixels.

Randomly spread across the surface, the fiducial markers 609 arerelatively scarce occurrences, covering only a small part of the totalroad surface 604. In at least some embodiments, the fiducial markers 609are (statistically) widely spaced apart from each other so that thereare significant dark areas around each illuminated fiducial marker. Thefiducial markers reflect brightly and are easy to detect. Filtering atthe camera, and optionally in the material or construction of reflectingfiducial markers themselves, greatly favors the reflection and detectionof illumination wavelength.

For example, when a single narrowband light source, such as a 405 nmVioletBlue (“DeepBlue”) or a 445 nm Blue laser beam, is used forscanning there are methods, such as described in U.S. Pat. No. 8,282,222and other references cited above, which ensure that only the laserwavelength is reflected. Furthermore, in certain retroreflectors, suchas cubic or hexagonal reflections (described, for example, in U.S. Pat.Nos. 9,810,913 and 9,946,076, both of which are incorporated herein byreference in their entirety), the reflected light is highly concentratedand observable only within a very narrow retro-reflection cone which iscentered on the incident ray (coming from the scanning light source)back toward the light source. Thus, it can be arranged that suchretro-reflective markers 609 would be easily detected even at largerdistances and in full daylight by sensors placed in close proximity tothe scanning laser source.

Further, as described in, for example, U.S. Pat. No. 9,753,126 and otherreferences cited above, in at least some embodiments the camera 604 maybe configured so that the sensing pixels of the camera are sequentiallyactivated and synchronous with the anticipated motion of the scanningbeam of light, so that any such reflections are preferentially receivedwhile spurious ambient light and other unwanted signals arepre-emptively filtered out.

In at least some embodiments, one or more of three complementary methodscan be used to filter out or disambiguate the system's scanning signalreflections from ambient and other spurious noise. First, spatialfiltering methods, such as using reflective and refractive optics, canbe employed. In at least some embodiments, by using retro-reflectingfiducial markers 609, the transmitted beam is only seen returning at ornear the location of the light source. On arrival of thisretro-reflected light, the optical system (for example, the lens) of thecamera 604 can further sort according to incoming direction, matchingeach incoming chief ray direction with a specific pixel.

Second, temporal filtering methods (for example, signal timing orgating) can be used. Because the scanning beam of light moves in aknown, observable, time sequential pattern, selective synchronizedsequential activation of one or just a small block of pixels highlyfavors reflected light arriving in that area over a very narrow (forexample, microsecond) time slot.

Third, wavelength filtering methods (for example, narrow band passfilters or special reflection coatings, such as Bragg coatings) can beused. In at least some embodiments, the light source can be narrow bandand can be matched by a narrow band pass structure in one or both of theretro-reflective markers 609 or the camera 604.

As an example, a 445 nm Blue laser source is highly collimated andscanned, for example, in a 1D line scan or a 2D pixel scan pattern,across the field of view illuminating small portions of the field ofview in rapid succession. Retro-reflecting markers embedded in thesurface selectively reflect just one wavelength of the light back in thedirection of the light source. There may be multiple light sources suchas, for example, one light source in each of two or more headlightassemblies. On arrival at the camera, the retro-reflected narrow bandlight passes through a narrow-band pass filter, and is collimated to aspecific pixel in the camera. Using the known spatio-temporal scanpattern of the scanning light source, the receiving pixels of the cameraare activated just before the light arrives and deactivated immediatelyafterwards. The latter may be automatic, (for example, theself-quenching logic found in SPADs can be added to gated receiverarrays of all types (rolling shutter, twitchy pixel or SPAD array)).When, if any, of the reflected light illuminates the camera, even whenthe signal amounts to just 10 photons in the case of SPAD receivers, theretro-reflected light will be detected. Through any combination of oneor more of the filtering methods described above, the selectivity ofthis system is such that a relatively small amount of reflected light(for example, retinal safe low power beam generated by a blue lasersource) is sufficient for observation even in full sunlight, with asignal to noise of 1,000,000 to 1 or greater advantage over sun lightfrom ambient sources or other system's light sources.

In at least some embodiments, the systems are arranged so that the beamsof light from the transporter (e.g., vehicle) scan the road surfacedirectly ahead (for example, the volume of 3D space in its plannedtrajectory) and the system determines possible alternative paths in realtime (for example, a multi path arrangement) while monitoring other roadusers and possible obstructions and hazards such as cars, pedestrians,bicycles, dogs, road debris, tumble weeds, and the like. In at leastsome embodiments, this system calculates and re-calculates one or morepossible safe paths around obstacles while checking the transporter'scurrent six DoF system kinetics, velocity, acceleration, pitch, roll &yaw, and the like and, in some embodiments, may consult availabletelematics on the state of its road hugging system (for example, howwell the wheels are each gripping their piece of the road right now andwhat is the prognosis for the next 30 feet ahead). In at least someembodiments, the state of road ahead is described by an update of therut, grip, bump, wetness and oil slick on a map (such as a detailed roadcondition or micro-obstacle map) which may be provided via V2X(vehicle-to-everything), V2V (vehicle-to-vehicle, which may be updatedjust minutes ago by previous cars in the lane) or perhaps at the outsetof each commute downloaded by the charging system, or, in the case ofmobility services, at a recharging or fueling station, or as part of anen-route, on-the-go energy and data provisioning along specificcorridors installed in frequently traveled road grid sections by theconsortium of mobility transport service providers or by any othersuitable arrangement or mechanism.

In at least some embodiments, a laser beam or other light source of thesystem can illuminate a few fiducial markers which are arranged in aspecific sequence (for example, green-blue-red-green-red-blue-blue, seeFIG. 16) where each marker is unique, surrounded by a suitable maskingtar or road surface colored paint or adhesive. Each marker isretro-reflective and brightly reflects the white or RGB (red-green-blue)laser beam in its specific primary color. The stereo or trifocal (ormulti camera) system instantly computes the 3D positions of the observedfiducial markers. In at least some embodiments, the time of transitionis also noted, for example, using a nanosecond precision clock. Evenwhen using a rolling shutter system the system can match the observed 3Dpositions of the fiducial markers in its FoV from the vehicle's machinevision Ego-perspective; that is, when viewed from the vehicle's viewthere are N colored fiducial points seen ahead where the road should be.These points should fit a surface where, when connecting the points withgraphs, the resulting polygonal mesh meshes (i.e., fits) the known (orprior provided) 3D fiducial map.

In at least some embodiments, if some fiducial markers were missed orlost by normal wear and tear of the road surface there is a simplesufficiency (for example, a number) of observations (i.e., a hammingdistance) where uniqueness of the sequence of detected colors and theirpositions in the surface will achieve a level of statistical certaintysuitable for the navigational task at hand.

As the road surface wears out, this would be detected automatically, andperiodically more fiducial markers can be added as a repair, or wholesections may be resurfaced as roads frequently are already. Worn outareas might be resurfaced by applying a fresh thin layer (for example, 3mm) of grit in a tar-like dark adhesive matrix material with randomlydistributed fiducial markers. Maps may be updated, adding portions withnew fiducial markers in the resurfaced road segments.

Once the observed color pattern has been fitted onto the detailed priormap, the vehicle navigation system knows exactly where it is. In atleast some embodiments, the degree of fit may be an indicator ofcertainty and of the quality or statistical accuracy of the derivedinformation. In at least some embodiments, each observation iscorrelated with a microsecond observational time, since the scanner's(i.e., light source's) position follows a known periodic function. In atleast some embodiments, the observed scan trajectory can be continuouslyre-calibrated each time the beam illuminates a known fiducial locationembedded in the road's surface. In at least some embodiments, the 3Dtrajectory of the vehicle and its six DoF dynamics can be estimated to ahigh degree of precision. For example, the vehicle's drift or cant(e.g., leaning, pitch, or roll) during steep turns can be nearlyinstantaneously detected and, at least in some embodiments, correctedusing, for example, by an active suspension system.

In at least some embodiments, the system and vehicle will be able toknow exactly (for example, to 1 cm in 3 dimensions) where it is, whereit is heading, and what its current “drift” or rotation, yaw and pitchis with respect to the main motion trajectory. In at least someembodiments, the system or vehicle can detect minute changes at thebeginning of skids, roll, bounces, or the like, and may correct forthese nearly instantly as appropriate, enabling pin point precisemicro-steering, a feature that is possibly life saving for CollisionImminent Steering (CIS).

The reflections of beam delays can be used as additional ToF measures.If the beam from the light sources scans fast, and the retro-reflectivefiducial markers are recognized and their known positions are looked upby the navigation system, then further refinements can be made becauseof the elapsed time of flight (“ToF”) after the illumination by thebeam. For example, the moment of exact illumination can be deduced fromthe scan pattern and the distance as described, for example, in U.S.Pat. No. 10,067,230, incorporated herein by reference. As an example, ifthe beam scans such a position in approximately 10 nanoseconds (forexample, a 20 microsecond line sweep with 2000 potential positions) thenthe observational moment per each fiducial marker would be retarded bythe distance from the fiducial back to the camera. For example, at 100m, the fiducial marker spatial registration would be delayed by 300nanoseconds.

A pixel sequential perception system (or line sequential perceptionsystem) can scan, learn, and then recognize every square mm of a road.This system can see both the location of a voxel and the color or greyscale (reflectivity) of that surface in nanoseconds. For example, a 25kHz resonant MEMS scanning mirror moves an illumination laser spot, froma light source, scanning at 50,000 lines a second across a 2 meter wideswath of the road ahead. If the laser spot scans the road over voxelsthat are 1 mm in diameter, in 20 microseconds the laser spot movesacross 2 meters of the road ahead. The tip of the beam traverses theroad rapidly back and forth, sweeping orthogonally to the direction oftravel.

The light source might only scan in a single direction, for example, inthe lateral direction (horizontally from the camera's perspective,across the pavement in front of or below the vehicle), or it may also becapable of a longitudinal deflection (vertically from the sensor'sperspective, along the trajectory ahead). In either case, in at leastsome embodiments, successive lines can be scanned at 50,000 lines persecond.

If the longitudinal (vertical) direction scan system illuminates 1000line segments of 1 mm by 2000 mm, scanning 2 meters of successive andadjacent stripes across the road ahead, the system in a vehicle 750 hasscanned every square mm in a 2 square meter section 753 of the roadahead, as illustrated in FIGS. 7A and 7B. Every one of these 2 millionsquare millimeters has been illuminated individually for approximately10 nanoseconds, taking a total of 20 milliseconds to scan this one 1×2meter section of the road ahead. Clearly if coupled (or synchronized)with a 2 Megapixel rolling shutter camera, the system can scan the roadahead in a fine grained detail at speeds up to 50 meters per second (180km per hour, approximately 111 mph).

A drivable road surface has been mapped using the systems describedabove. In at least some embodiments, the resulting map can contain bothvoxel location information (for example, a mm accurate 3D profile of thesurface) and pixel contrast information (for example, a mm accurateimage with all contrast visible details of the road grid.) This may beuseful as the hard crushed aggregate embedded in the tar matrix makesthe road surface non-slip and can be detected in the pixel contrastinformation. The map also contains markers such as road paint, cat-eyereflectors, and fiducial markers as described above. Further, the mapmay contain any other location information pertinent to the roadcondition, such as, for example, rutting, likelihood of hydroplaning,other wear conditions, or the likelihood of the formation of black ice.This kind of ultra-local road information may be based on actualincidences or accidents, or observations by vehicles ahead and, in atleast some embodiments, may added to the map as specific localconditions nearly in real time.

In some embodiments, the map may provide precise details about thesurface ahead such as the slope or grittiness (for example, the surfaceroughness, bumps, or the like). The cant of the road of the drivablesurface can be known by the system (for example, an autonomousnavigation system) well ahead of any required maneuver, whether routineprogression during a commute or just in case of emergency evasion orcollision avoidance. The system can be always aware of the road ahead,with kilometers of road surface data loaded in active system memory. Aterabyte of flash memory, the amount available in the most recent mobilephones, could hold a mm accurate map of a 1000 km road.

Given the vehicle characteristics, the road characteristics and thetraffic situation the vehicle can choose the appropriate speed andcreate an optical “flight plan” for the road ahead, analogous to theflight plan submitted by airline crew ahead of a flight that takes intoaccount wind conditions, weather and other traffic.

As an example, the system may adjust the height of the suspension forthe smoothness of the road ahead, and take into account the windconditions, required stability in steep turns, the vehicle's loading, orthe like. Anticipating some bumps the system can adjust the height ofthe carriage briefly upwards smoothly gliding over bumps, such as bumpytransitions that often occur at bridges or when part of the road is inprogress of being re-surfaced. This “flight plan” 755 can be a 3Dtrajectory for the vehicle's main cabin to “fly” or “glide” and may bemuch smoother than the trajectory of the vehicle's under-carriage whichthe wheels actually follow.

Thus the wheels individually and as an under carriage functionmechanically to maintain an optimal grip on the road, whereas thesensitive cargo or passengers experience an optimal smooth ride, in a“glide path” that is planned and executed by the system.

Many multi-axle vehicles re-tract and off-load each of their wheels justan instant before stepping up and over road surface defects andre-engage with the better surface afterwards. In electrical vehicles(EVs) the weight of the battery pack typically outweighs all othersystem components. As an example, in a Tesla model 3, 30% of the totalvehicle weight or approximately 1000 pounds (about 454 kg) is the 100kWh battery pack. In ultra-light single person vehicles using advancedaircraft materials or novel ultra-light and ultra-strong laser sinteredtitanium honeycomb structural frames, the ratio may be even moreextreme.

Further, on novel high-speed long distance AEV (autonomous electricvehicle) highway lanes, great speeds (180 km per hour) can be achievedover great distances with a wireless energy transfer to the vehiclethrough the driving surface. For maximum speed, stability and powertransfer efficiency it is desirable that the vehicle's frame, whichcarries the batteries, “hugs the road” as close as possible. Thus itwill by necessity experience at least some of the unavoidable roadimperfections. There is no need for the passenger to experience these.

Using inertial sensors in the cabin floor or the passenger seat, anactive suspension system can anticipate each of the road trajectory'sbumps and effectively cancels it out. In at least some embodiments, asillustrated in FIG. 8A, there are two suspension systems, one system forthe undercarriage 860 with the batteries, power system, and wheels, anda separate system 861 above the first system to smooth out the remainderof bumps and motor and steering effects for the cabin section 862(passenger compartment). In some embodiments, the passenger compartmentmay include optional ailerons 864 or a rudder 866. The second suspensionsystem 861 can be active and react to changes in the road surface. Atether 868 may couple the cabin section 862 to the undercarriage 860. Inat least some embodiments, this arrangement can provide an ultra-quietcabin and enable the passenger(s) to sleep even when cruising at 300 kmper hour on straight stretches of recently upgraded “super AEVhighways”.

FIG. 8B shows another example of an undercarriage 860 and cabin section862. In at least some embodiments, an inductive magnetic charging systemis embedded in an ultra-fast AEV lane adding an electromagnetic downforce to the undercarriage 860 ensuring road adhesion for the wheels ofthe undercarriage to better grip the road. In addition, rather thanusing an active suspension the cabin section 862 may just separate (butheld by the tether 868), and literally fly above and behind the undercarriage using ailerons 864 and a small rudder 866 to smooth out theride along the flight path just above the road surface.

According to JIEDDO, the Joint IED Defeat Organization (USA Todayreported on Dec. 19, 2013, as told by Thomas Friedman in his book “ThankYou for Being Late”) more than half the Americans killed (3,100) orwounded (33,000) in the Iraq and Afghanistan wars have been victims ofIEDs (Improvised Explosive Devices) planted in the ground.

As illustrated in FIG. 9, in addition to the vehicle 950, a remotecontrolled or autonomous vehicle 951 (for example, a quadcopter or adrone) can move along a paved road 908 scanning the road in its path atspeeds up to 50 meters per second (more than 100 miles per hour) usingthe systems and methods described above. This second vehicle 951 maynever touch the road, but flies above the pavement at speeds of, forexample, 50 meters per second, around, for example, 1 meter or moreabove the pavement. Recording and comparing the surface daily for evenminute changes will confirm that the surface of the pavement andsurrounding banks of dirt were truly undisturbed. Thus, the currentstate of the pavement is compared to prior observations to ensure it hasnot been tampered with. Roads could be paved with a coat of pseudorandomly placed fiducial markers to prevent counterfeiting.

One advantage of the systems is that only the pavement is scanned ratherthan the whole environment. Other systems, such as Road DNA™ by TomTom™,scan the whole ambient environment of the vehicle as it would be seen bydrivers. The RoadDNA™ system constructs a map of walls, buildings, andother structures that line the road. To construct this map, the privacyof homes, gardens and private spaces of people is unavoidably invaded.The company says it removes such acquired information. However thisimplies that large sections would be redacted out.

By contrast, the systems described above only use public surfaces (orprivate road surfaces) which typically contain no private information.Therefore, there is no privacy invasion. The system only sees roadsurfaces it drives on and, optionally, any vehicles or obstructions onthe road. The system may map your private driveway's surface but thisinformation can be kept off line, if so desired.

Light beams from light sources can enable Fast Foveated Perception(FFP). A vehicle is equipped with two scanning headlight assemblies. Ineach headlight assembly there is at least one camera. Preferably, thecamera projection center is substantially co-located with the projectioncenter of the scanning headlight. The camera may be mounted with a relaymirror that closely (substantially) aligns the optical axis of cameraoptics, i.e. the camera's FoV, with the FoV scanned by the headlight. Inthis exemplary case illustrated in FIG. 10 there are two headlightassemblies and two cameras having sensor planes P_(L) and P_(R),respectively, and optical projection centers O_(CL) and OCR,respectively. In other embodiments, it is possible to have just onescanning headlight.

The two cameras have at least a partially overlapping area in theirindividuals FoV. This is the camera stereo field of view, FoV_(cs). Abright illumination spot S is projected by one (or both) of the scanningheadlights, and reflects off an object in the FoV_(cs) at a distance Zfrom the vehicle. The two cameras activate respective subset portions oftheir sensors that match the reflected images SL, SR of the spot S ineach of their sensors.

The spot illumination interval is very brief, for example, only 100microseconds. There may be, for example, up to 100 fractions making upthe total FoV_(cs) which the system can illuminate individually,selectively and sequentially. One such fraction, at a scanning framerate of 100 fps, is illuminated and exposed for 100 microseconds: a1/100^(th) fraction of a full frame exposure of 10 msec.

The activation (shuttering) of subareas in the sensors might be equallybrief because an intelligent, dynamic, illumination control system cananticipate and selectively activate the location of the foveated light'sreflected image in the sensor, and sequentially move that location inthe sensor along a trajectory that matches the flying illuminationspot's trajectory across the FoV.

In at least some embodiments, a light source (or a pair of lightsources) selectively illuminate the spot S and a pair of stereo camerasonly send a small subset of exposed pixels from each camera back to animage processing system or a machine vision system due to the selectiveactivation of camera pixels. This not only increases contrast anddecreases acquisition time, it further significantly lowers the totalsystem latency by reducing the total amount of pixels that need to betransmitted, resulting in a smoother stream of nearly blur free pixels.Furthermore, it reduces the computational complexity, and therefore theautonomous driving system, or collision avoidance system, can be moreagile and respond faster to, for example, hard-to-detect small, darkroad debris.

In at least some embodiments, a smart active illuminated machine visionsystem can activate a group (for example, a bundle, ribbon or band) ofrows in a conventional rolling shutter camera sensor. Rolling shuttersensors are quite inexpensive and have already been deployed as camerasin mobile phones. For example, a mobile facing camera may have 2 millionpixels arranged in 2000 columns and 1000 rows. In such a camera arolling band of 100 rows ( 1/10^(th) portion of the whole sensor) can beselectively exposed (reset and activated) and read out. Further, asillustrated in FIG. 11, the system may select to read and transmit onlypixel data from a subset of rows 1174 and a subset of columns 1175 (forexample, the area of interest 1176).

In at least some embodiments, circuitry added to the camera Analog toDigital (A2D) decoder can select just pixels with exposures above(and/or below) a certain dynamic signal threshold. These thresholds maybe set to ensure that objects within a certain range are recorded. Dueto the steep drop off (typically 1/r²) a small and diminishing fractionof the scanning spot light is reflected by objects and surfaces atlarger distances. Effectively then only those objects or surfaces withina certain range reflect enough light to be captured by the camera'saperture, and only those are recorded after illumination by aknown-to-be-sufficient strobe illumination level energy.

This illumination level and the sensors' threshold settings can becontrolled by the same system. The selected pixel data are transferredfor further processing in the sensor's local circuitry, or downstreamusing, for example, perceptual algorithms or AI functions (for example,CNN—convoluted neural network) which parse and crop, associating certainselect pixels as groups into objects and then attempt to assign themclasses, adding object classification labels such as vehicle, bicycle,pedestrian, or the like.

In at least some embodiments, there can also be a maximum exposurethreshold, to flag over-illuminated objects or surfaces in theforeground. Such over-illuminated, over-exposed, or saturated pixelswere most likely too close and might already have been processed andclassified in earlier frames. This function would also help ignore (forexample, mask out) raindrops and snowflakes, removing them at thismoment. The latter function might help keep snow and rain from occludingobjects of interest.

For example, in FIG. 12, raindrops 1280 are masked out as the camera1204 sees over-exposed pixels 1282 due to retro-reflecting a smallportion of the light of the light source 1206 and due to the proximityof the raindrops in the foreground. The raindrops are effectivelyremoved from the image by the system, keeping the raindrops from addingunnecessary ambiguity to the machine vision system examining the objectof interest at a farther distance.

Turning to FIGS. 13A-13C, a headlight can include a light source 1306, aphosphor 1386 and a recollimator 1387. The light emitted by theheadlight, has two components: a first portion 1383, about 20% of it,consists of the diffused and incoherent form of the original shortwavelength coherent laser source 1306, for example, a 445 nm laser. Therest 1384 consists of Stokes shifted (absorbed and re-emitted)fluorescent photons of a range of longer wavelengths across the spectrum(blue, cyan, green, yellow and red). This fluorescence energy conversiontakes time. The longer wavelengths emitted by the phosphor 1386 emergewith a delay after the blue higher energy photons have been absorbed bythe phosphor. Circuitry 1388 in the sensor can be arranged to exploitthis delay so that the first to arrive blue (for example, narrow bandlaser light 445 nm) photons trigger adjacent, nearby RGB (for example,Bayer pattern) filtered pixels (labeled “R”, “G”, and “B” in FIG. 13C)just as the delayed broad band wave arrives. FIG. 13B shows an the firstto arrive narrow band pulse 1383 trailed by a later pulse 1384 of broadband light of longer wavelengths. Narrow band pass (for example, Braggtype) filters on the photodiodes that act as triggers effectively canshield out the ambient light. Since the broad band (RGB) pixels are onlyactivated for a short duration of the broadband pulse this just-intime-trigger function can effectively block out ambient light.

Raindrops retro-reflect, reflecting light back extra brightly towardsthe source of the light. Therefore, having two lights set wide apart andhaving the opposing camera look at objects of interest the FoV_(cs)helps not being blinded. For example, as illustrated in FIG. 14, theleft camera 1404 looks at object illuminated by right beam 1406, andvice versa.

Alternating stereo cross illumination (ASCI) helps the computer visionsystem not being blinded by the head lights, for example, in the case ofrain or snow, and also would help mitigate the blinding effect ofconspicuity type retro-reflective markers such as cat eye reflectors andother retro-reflectors that are found everywhere on and around roads(for example, on the rear of most vehicles, on clothing, helmets andshoes, on traffic signs, and nearly all road surface markings).

In at least some embodiments, the system may choose to focus (forexample, selectively foveate) on these retro-reflective (RR) type ofmarkers using a scan cycle with a lower light intensity with anappropriately high set threshold to spot only these exceptionally brightretro-reflective markers. The system alternates between such “spotmarking RR” cycles (specifically targeting RR type markers) and “maskingRR” cycles (specifically ignoring RR type markers), and then later onduring downstream post-processing combines two such alternating scans inrapid succession. This form of data fusion enables the system tomaintain a dynamic map that tracks the current traffic environmentaround the vehicle.

Bright retro reflectors are easily and quickly spot-marked in specificdedicated “spot marking RR” scans, and later these RR positions thenassist image capture as they are recognized and noted in alternatinghigh contrast image capture cycles. For example, as illustrated in FIG.15, four bright spots 1590 in a “spot marking RR” scan 1592 cancorrespond to the vertices on the back of a truck 1550 identified in a“masking RR” scan 1593.

In at least some embodiments, the system can include a laser spot RRmarking process. Firstly, the system marks only those pixels brightenough to exceed the minimum illumination threshold (such as foveatingin the far field—for example, farther than 30 meters away). In a secondcycle, the system notes (not as grey scale value, bur rather a binary RRflag set and associate it with that pixel) the pixels exceeding amaximum exposure threshold and matches them up with pixels (for example,pixel locations in the FOV_(cs) of the previous observations). This(Min-Max cycle) method would help to very accurately find, position andtrack the bright RR marked features and associate these fiducialfeatures correctly with less conspicuous (non-RR) features of theobjects they belong to. For example, in FIG. 15, the reflectors 1590 onthe rear of a truck or, in other example, signs with RR and non-RRmarkings.

In at least some embodiments, a Blue laser light may be used selectivelyto spot-mark the RR markers using dedicated blue (for example, 405 nm)narrowband filtered cameras. Simultaneously, regular RGB cameras mightobserve the same view blocking out the laser primary with a narrow-bandblocking filter. The advantage is that the fiducial markers can betracked this way easily in daylight and at night using very littleenergy while the RGB cameras can use light from ambient sources and/orjust light provided by the vehicles from broad-spectrum headlights toobserve the road surface and objects on the road.

Embodiments, however, are not limited to only one or two cameras orlight sources. It may be advantageous to have three or more camerasilluminated by a plurality of light sources, analogous to examplesdescribed in, for example, U.S. Patent Application Publication No.2018/0180733, incorporated herein by reference in its entirety. In thismultiview architecture. a laser pointer or laser brush illuminates atrajectory of individual voxels with a rapidly scanning “pin prick”laser illumination beam, and the series of illuminated voxels areobserved with three or more cameras.

In at least some embodiments, a robot delivery vehicle (or any othersuitable vehicle) might have a third top mounted camera that would bedeliberately positioned far away from the lower mounted stereo pair ofheadlights and cameras. The third camera would be less susceptible tobeing blinded by retro-reflectors illuminated by the actively scanningheadlights below. A further advantage is that it could look father aheadfrom its elevated position, looking over the top of other vehicles andobstacles.

A third scanning illuminator could also help in looking for doorhandles, address signs and potential obstructions overhead. It mayilluminate obstacles such as trees & shrubs, and enable the system tospot their branches and other protrusions and obstacles jutting across avehicle's path along, for example, a narrow route.

In FIG. 16A, two or more projectors (i.e., light sources) 1606 areconfigured to illuminate, and two or more cameras 1604 are configured toobserve, a common (at least partially overlapping or shared) field ofview 1607. The projectors (i.e., light sources) are scanning typeprojectors, preferably highly collimated in a narrow illumination beam.Optionally, the light is generated by a Chrystal Phosphor Fluorescentsource that converts at least part of an incident intense blue or UVlaser source into a broad-spectrum (“white”) illumination. But it mayalso be a diffused or wider highly collimated monochrome beam. Theillumination creates a spot on remote surfaces and objects. Thereflection of this spot is observable by the cameras 1604 in eachcamera-projector pair, as well as by at least one of the other camerasin the system. Ideally, the cameras and projectors in eachcamera-projector pair are substantially co-located, i.e. they share thesame perspectives and share their elevation and azimuth rotationcoordinates of the illumination.

FIG. 16B illustrate an arrangement with one light source 1606 and twocameras C₁, C₂. It will be understood in other embodiments, there may beseparate light source associate with each camera. FIG. 16B alsoillustrates a scanning mechanism 1620, reflector/phosphor 1621, andcollimator 1622. The light source 1606 illuminates a spot S on a remoteobject. The camera C₁ of observes and records an image caused by thereflection of this spot S′ in its sensor as a group of pixelsP_(1ij)-P_(1kl). The camera C₂ also observes the same spot S asreflected onto its sensor as spot S″ in the form of a second group ofpixels P_(2mn)-P_(2op). This spot will be in a different position.

A processor finds correspondences between the two spots. There is, forexample, a feature F on the object illuminated by the spot resulting inits image being projected as F′ in camera C1 and as F″ in camera C2. Thematching of F′ with F″ is the result of a search of all pointsP_(1ij)-P_(1kl) in C1 and comparing them against all pointsP_(2mn)-P_(2op) in C2.

So, for example, if the spot images S′ as 1,000 pixels and it images inan approximately equal number of pixels as S″ a feature matchingprocedure may examine at most 1,000,000 possible pairs of featuresF′-F″s where F′ belongs to P_(1ij)-P_(1kl) in C₁, and F″ belongs toP_(2mn)-P_(2op) in C₂.

In at least some embodiments, FIG. 17 illustrates an augmented reality(AR) system that includes a head mounted display (HMD) 1730 which tracksthe eye gaze of the user. The user gazes upon a close object, forexample, her fingertip T. The HMD system tracks the gaze of her eyesprecisely, so as to be able to position (“anchor”) augmented realityenhancements affixed to her finger, for example, a small virtual flame1732 flickering from her fingertip T. Two cameras, C₁ and C₂ arepositioned on the left and right side of her wearable HMD, for example,on the hinges of her AR glasses observing the field of view in front ofher head. The observable field of view scanned by the cameras may bequite wide, for example, 120 degrees wide in total and 90 degreesvertically with a substantial stereo perspective overlap between thecameras, for example, approximately 70 degrees in the far field (itwould be less in the near field closer to the user's head). Each of thecameras might scan 100 degrees, with approximately 10 degree monoscopicperspectives both on the right and the left side of the 120-degree FoV.In the stereo overlapping FoV area, any place the user might foveate iscould have a resolution of at least 20/20 human vision, which can be,for example, 60 columns (pixels) per degree. At that resolution each ofthe cameras would have 32.4 Mpixels (90 degrees by 100 degrees, or90,000″ square degrees at 3600 pixels per square degree).

It is advantageous to use a short wavelength to scan the world so thatthe size of the camera sensors can be kept relatively small. This is whyusing blue laser light, for example, a wavelength of 445 nm, to scan thereality to be augmented is desirable. It enables a pixel size as smallas 500 nm, and this keeps the sensor economical—as the active pixelarray of the 32.4-megapixel sensors 6000 columns and 5400 rows can havea size of 3 mm by 2.7 mm. This reduced size allows the cameras to befitted on the HMD, yet still have sufficient spatial resolution whenusing short wave length (blue) illumination.

Another challenge is the inherent energy requirements and latency issuesassociated with scanning across such a large array. The solution is tofollow the user's gaze, and “co-foveate”, that is, to focus and restrictthe sensor activation and read-out to a small subsection thatcorresponds to the user's gaze. When the user is staring at her fingertip, like her eyes, the HMD cameras need to critically resolve only“what she is staring at”, i.e. only her fingertip. Any other sections inher field of view are much less critical to perceive, or to anchor.However in the area of her foveation, 3D spatial perception is ofimportance.

HMDs can track the user's gaze to fractions of a degree. Using thissignal the system can select and activate, for example, only a 90,000pixel subsection of the 32.4 M Pixel sensor. This is a very smallfraction of the array, less than 0.3%. Foveation of the camerasfollowing the user's gaze greatly reduces the computationalrequirements, the energy, and the latency of the HMD perceptual system.

Tracking the user's gaze can equally drive the auto focus of the HMDcameras. Typically, the accommodation is driven by the vergence detectedas the “towing in” of the user's eyes. Looking at the fingertip, theuser's left and right eye gazes converge and her lenses accommodate aspart of her foveation on her fingertip. Detecting that close upvergence, and the exact location, her AR device's cameras follow herfoveation on her finger. Because the position of these HMD camerassubstantially differs from the position of her eyes the vergence andaccommodation of the cameras is different, but can be calibrated as aprogrammable geometric algorithm.

Optionally, the HMD might also be provided with scanning illuminationbeams that track the gaze and selectively illuminate the fingertip,“catching” her fingertip in a spot light, illuminating it selectivelywith brief flashes of laser light.

Optionally, strobed flashes of bright blue (445 nm) laser light triggercan select subsections in the sensors as described above.

Further, optionally, the light beams illuminating her finger—the objectof attention she and her device are foveating on—mark a fiducial mark onher finger, adding, for example, a sharp instant contrast that enablethe cameras to align their gaze at pixel resolution and thus greatlyfacilitate stereo matching (feature pairing) of the two views of thefingertip.

In at least some embodiments, the illumination beam selects a smallsubset of the field of view. Further, in the illumination is astructured fiducial pattern, for example, a cross-hair pattern, thatserves the function of allowing cameras with different, potentiallypartial, perspectives of the illuminated manifold areas to quickly findat least one matching point pair (N-tuple, if there are N camerasobserving the illuminated manifold area) which then serves as a “seed”to finding other correspondences in areas adjacent to that fiducialreference.

FIG. 18 illustrates one example of retro-reflective fiducial markersthat are embedded in a road surface. FIG. 18 is a camera perspective ofthe surface. Its shows only a small subsection 1860 of approximately 72positions in the camera sensor and corresponding road surface (e.g.,twelve columns, #109 to #120, and six rows, #511 to #516.) The firstgreen fiducial marker is detected in row #511, positioned at column #115and is illuminated at a time “10230 nanoseconds” (the latter may just bean arbitrary clock timer incrementing every 10 nanoseconds). The nextfiducial marker is detected two rows down (later) and is blue at row#513 and column #112.

It will be understood that perspectives and the scanning sequence mayvary but the absolute geometric distance is invariant, and can berecorded and recognized. For example, the distance from G₁ to B₁ on thesurface is approximately 36 mm if the unit scale in the sensor is 10 mmper pixel as it is observed two rows down (20 mm longitudinal direction)and three columns sooner in the scan (30 mm in the lateral direction)and then the absolute distance is approximately 36 mm.

Distances and colors of adjacent RR fiducial markers thus create aunique spatial color mesh which can be recognized and which uniquelydefines a certain position on the road as well as a perspective of anobserving camera. Keeping track of the precise temporal sequence enablesthe motion and kinetics of the observer to be very accurately determined(for example, at least centimeter spatial precision at 10 nanosecondtime precision). The spatial distribution need not be very dense and 3colors would suffice, but more or fewer color may be used. With exactspatial coding, a monochrome system using clear RR fiducial markers mayalso work sufficiently well.

FIG. 19 illustrates an optical receiver or camera 1904, with an array ofpixels 1905 (one of which is expanded) acting as real-time locationdetector. Pixels “twitch” when the photodiode 1965 receive lightreflecting from a spot projected by a laser on a remote surface.Electronic circuits (transistors, digital or analog logic, for examplethreshold and amplify logic 1961 i, 1961 j) for individual pixels in thearray are configured to send the pixels location, for example, thepixel's row and column address, to a processing system as soon as thephotodiode in the pixel receives (i.e., detects) a photon flux intensityexceeding a minimal threshold value.

In at least some embodiments, the sensor, circuit and optics can betuned to be selectively, optimally or exclusively sensitive to a singlewavelength, for example, 405 nm (as very short wavelength, near UVwavelength which to the human eye in sub 10 mW intensities can bescanned both safely and nearly invisibly.) A twitchy pixel sensor may besensitive to broader bands of light, or may be tuned to other visible orinvisible wavelengths such as NIR (for example, 850 nm or 940 nm) or MIR(for example, 1550 nm)

A simple sensor may have, for example, 1000 rows of 1000 pixels, whereeach pixel has a small standard CMOS photodiode coupled to high-gainamplifiers, with sufficient drive to pulse both a row and a columndetection sensing line. The “twitchy pixel” circuits acts as binary(on/off) spot detectors which are by default off. At the periphery ofthe pixel array area, 1000 row and 1000 column lines are connected to asimple address encoder. When an individual row or column is triggered,the encoders translate (encode) the identity of such triggered (pulseactivated) lines in real time by sending a corresponding digitaladdress. Each of 1000 columns and of 1000 rows can be encoded in a10-bit address.

These row and column address encoder circuits may have two separatededicated serial outputs 1971. They nearly simultaneously send out a10-bit column address (indicating the x location of the pixel in thearray) and a corresponding 10-bit row address (indicating the y locationof the pixel in the array). In this configuration, the pixel address canbe sent to and processed by a processor, as soon as the pixel detects aphoton flux exceeding the set threshold. Thus, the scanned trajectory ofthe laser results in a steady stream of pixel locations being reported,each representing a sequentially observed—distinct-laser spot location.In at least some embodiments, detected fiducial marker locations arereported in real time, each sequentially, to a processing system withminimal latency within a few tens of nanoseconds.

Optionally, both X and Y locations are sent sequentially in a 20-bitstring on the same serial line 1973. See, for example, U.S. Pat. No.9,581,883 and U.S. patent application Ser. No. 15/853,783, both of whichare incorporated herein by reference in their entireties.

Optionally, the photodiode detector may be an avalanche photodiode (APD)or silicon photomultiplier (SiPM) array provided with analogousreal-time, low-latency column and or row address encoding circuits thatprovide the location of each successive avalanching pixel event.

FIG. 20A illustrates a retro-reflective transparent sphere S (i.e., afiducial marker as describe above) that can be added to an existing roadsurface (ERS) 2008 by mixing the spheres with an adhesive coating (AC)layer 2009 which is painted or sprayed onto the existing road surfaceERS, where it dries and hardens. The retro-reflective spheres end uprandomly or irregularly distributed (or distributed in a regulararrangement, if desired) and fixed to the road. This creates a uniquepattern that then can be used as a highly accurate positioning system byvehicles with the systems described above.

Turning to FIG. 20B, before mixing with the adhesive coating, thespheres S may be coated with a hard-coated anti-reflective coating 2081which serves as a narrow bandpass filter. Light from the strobe or scanlasers will pass selectively through these narrowband pass filters up to4 times: first when entering, then twice when retro-reflectinginternally, and finally when departing. In at least some embodiments,outer reflective coatings 2082 may be applied to increase theretroflectivity for reflections. These coatings may be designed toharden in places where the adhesive coating covers them (for example, isin contact with them) serving to specularly reflect light within thesphere. But in the top driving surface the sphere's optical surfaceshould be exposed, and a portion 2083 of the reflective coating 2082might be designed to dissolve or vanish by, for example, a chemicalwash, UV exposure, or chemical components added to this layer effect theselective removal of the exposed portion 2083 of the reflective coating2082.

FIG. 20C illustrate shows that hard quartz grit (G) particles, which aretypically added to top layers coating drivable surfaces 2008, may bemixed with a sparse amount of spherical retro-reflective fiducialmarkers (S).

Wear and tear of the road can expose the hard glass colored fiducialmarkers, such as hard-coated scratch-resistant spherically shaped smallglass beads. The tar adhesive coating provides contrast with thefiducial markers. Its main function is to lock the fiducial markerspermanently in place. It could be composed of specially formulatedepoxy.

Note that as the surface wears out some markers inevitably will be lost,and a new topical coating patch with fresh fiducial markers can beapplied using standard road maintenance techniques and procedures. Thisprocess of maintenance can be automated, enabling a robotic repairsystem.

FIG. 21A illustrates a top view of a section of road surface, in whichat random places retro-reflective fiducial markers have been placed.Fiducial markers 2109 a mark a section 2108 a immediately ahead of thevehicle. A downward looking linear pixel array camera R sequentiallyobserves the road surface directly below the camera while it isilluminated by a bright strobe illumination scanline I thatsubstantially overlaps with the linear aperture of the camera R. In atleast some embodiments, the pixels in this array may be configured forasynchronous detection, as described in U.S. Pat. No. 8,696,141,incorporated herein by reference in its entirety, so that theobservational time might be accurate to a microsecond. The camera maybe, for example, a linear SiPM array (silicon photomultiplier array). Asresult in this arrangement the longitudinal travel velocity and positioncan be marked with precision.

The retro-reflection of fiducial marker 2109 b lights up pixel i in thearray. The system notes the pixel location at time t, and because itknows its current trajectory and velocity from prior observations, itcan confirm that this fiducial marker is at the expected location. Thus,the observation serves to update the vehicle's exact current location onthe road. The most recent observation joins a set of four or more priorsuch fiducial markers 2109 c observed and confirmed during the recentmillisecond in road section 2108 c. By known photogrammetry methods thevehicle's six DoF position and motion vectors can be updated. The systemcan provide 100's of such navigational and mapping updates per second.FIG. 21B is a side view of the light source S producing illuminationline I and the camera R. A fiducial marker 210 b is reflecting a portionof the illumination from source S back towards the camera R.

FIG. 22 illustrates an embodiment where two beams (B1 and B2) illuminatethe road surface below and cause successive retro reflections from twolines. These are observed by the same fast asynchronous linear receiver(camera) R. In this arrangement the distance of the receiver to the roadsurface 2208 (for example, the road clearance below the vehicle's frontbumper) can be tracked in real time. As shown in the drawing eachfiducial marker will be twice illuminated. Fiducial markers 1 and 2 areat distances z1 and z2 respectively from receiver R. Time t0 is the timeat which each of the fiducial markers is illuminated by B1 (and is notnecessarily the same absolute time for each fiducial marker). At a latertime t1, beam B2 illuminates fiducial marker 1, and causes a secondpulse at the same pixel location in the linear array. At yet a latertime t2 the second fiducial marker passes through beam 2, and itsinterval between successive pulses is correspondingly bigger. For anygiven vehicular velocity V, when Z2>Z1, then Δt1 (which equals t1−t0 forfiducial marker 1) is smaller than Δt2 (which equals t2−t0 for fiducialmarker 2). Accordingly, these time measurements can be used to estimatethe height of fiducial marker 2 relative to fiducial marker 1.

The asynchronous receiver “twitchy pixels” (e.g. APD or SiPM pixels)enables these relative time intervals to be observed with microsecondsprecision, and as a result accurate road clearance distances (front ofvehicle road clearance height) can be determined to, for example, the mmwithin a millisecond at 50 meters per second (112 mph).

The time interval observed between successive flashes is shorter as thereceiver comes closer to the pavement. This may help to reduce the roadclearance for optimal aerodynamics at high speed, for example, to enableimproved “road hugging.”, using a splitter or skirts to prevent air frombeing trapped below the vehicle. “Road Hugging” improves traction andsaves energy at greater highway speeds. In at least some embodiments,this arrangement can also be used to map the exact height with respectto the pavement or other fiducial markers. For example, it would allow amapping robot to sweep the road and determine its exact shape,curvature, and cant in 3D spatial coordinates.

What is claimed as new and desired to be protected by Letters Patent ofthe United States is:
 1. A method for navigating a vehicle along asurface, the method comprising: employing a scanner to scan a strobedlight beam over the surface, wherein the strobed light beam providesstrobed illumination of a plurality of lines across the surface toreduce motion blur, and wherein a stripe is formed by at least a portionof the plurality of lines having a width and a length across a field ofview on a portion of the surface that is directly ahead of the vehicle;employing light reflected by one or more fiducial markers on the surfaceonto pixels of a receiver to determine a spatial arrangement of thefiducial markers on the surface; comparing the spatial arrangement ofthe fiducial markers with a predetermined map of the fiducial markers todetermine a location of the vehicle; and detecting a reflection of thestripe from the portion of the surface with the receiver, wherein thestripe's reflection is employed to determine three-dimensional spatialinformation and visual contrast information that is added in real timeto the predetermined map for the portion of the surface that is directlyahead of the vehicle.
 2. The method of claim 1, wherein employing thescanner comprises employing the scanner to sequentially scan the lightbeam along a line or region of the surface.
 3. The method of claim 1,wherein the scanner is configured to simultaneously illuminate a lineacross the surface with the light beam, wherein employing the scannercomprises employing the scanner to sequentially scan a series of lineson the surface.
 4. The method of claim 1, wherein each of the fiducialmarkers has one of a plurality of different colors, wherein employinglight reflected by one or more fiducial markers comprises determining acolor of each of the one or more fiducial markers using the reflectedlight.
 5. The method of claim 1, further comprising employing lightreflected from an additional object on the surface onto pixels of thereceiver to determine a position of, or to identify, the additionalobject.
 6. The method of claim 5, further comprising determining a pathto avoid the additional object.
 7. The method of claim 1, furthercomprising employing light reflected from the surface onto pixels of thereceiver to determine condition of the surface.
 8. The method of claim7, further comprising adjusting operation of the vehicle in view of thedetermined condition of the surface.
 9. The method of claim 7, furthercomprising adjusting a suspension system in view of the determinedcondition of the surface.
 10. The method of claim 1, wherein employinglight comprises sequentially observing subsets of the pixels of thereceiver in coordination with the scanning of the light beam.
 11. Asystem to navigate a vehicle along a surface, comprising: a scannerconfigured to scan light over a field of view; a receiver that comprisesa plurality of pixels configured to detect light; one or more memorydevices that store instructions; and one or more processor devices thatexecute the stored instructions to perform actions, including: employinga scanner to scan a strobed light beam over the surface, wherein thestrobed light beam provides strobed illumination of a plurality of linesacross the surface to reduce motion blur, and wherein a stripe is formedby at least a portion of the plurality of lines having a width and alength across a field of view on a portion of the surface that isdirectly ahead of the vehicle; employing light reflected by one or morefiducial markers on the surface onto pixels of a receiver to determine aspatial arrangement of the fiducial markers on the surface; comparingthe spatial arrangement of the fiducial markers with a predetermined mapof the fiducial markers to determine a location of the vehicle; anddetecting a reflection of the stripe from the portion of the surfacewith the receiver, wherein the stripe's reflection is employed todetermine three-dimensional spatial information and visual contrastinformation that is added in real time to the predetermined map for theportion of the surface that is directly ahead of the vehicle.
 12. Thesystem of claim 11, wherein employing the scanner comprises employingthe scanner to sequentially scan the light beam along a line or regionof the surface.
 13. The system of claim 11, wherein the scanner isconfigured to simultaneously illuminate a line across the surface withthe light beam, wherein employing the scanner comprises employing thescanner to sequentially scan a series of lines on the surface.
 14. Thesystem of claim 11, wherein each of the fiducial markers has one of aplurality of different colors, wherein employing light reflected by oneor more fiducial markers comprises determining a color of each of theone or more fiducial markers using the reflected light.
 15. The systemof claim 11, wherein the actions further comprise employing lightreflected from an additional object on the surface onto pixels of thereceiver to determine a position of, or to identify, the additionalobject.
 16. The system of claim 15, wherein the actions further comprisedetermining a path to avoid the additional object.
 17. The system ofclaim 11, wherein the actions further comprise employing light reflectedfrom the surface onto pixels of the receiver to determine condition ofthe surface.
 18. The system of claim 17, wherein the actions furthercomprise adjusting operation of the vehicle in view of the determinedcondition of the surface.
 19. The system of claim 17, wherein theactions further comprise adjusting a suspension system in view of thedetermined condition of the surface.
 20. The system of claim 11, whereinemploying light comprises sequentially observing subsets of the pixelsof the receiver in coordination with the scanning of the light beam. 21.A non-transitory processor readable storage media that includesinstructions for navigating a vehicle along a surface, wherein executionof the instructions by one or more processor devices cause the one ormore processor devices to perform actions, comprising: employing ascanner to scan a strobed light beam over the surface, wherein thestrobed light beam provides strobed illumination of a plurality of linesacross the surface to reduce motion blur, and wherein a stripe is formedby at least a portion of the plurality of lines having a width and alength across a field of view on a portion of the surface that isdirectly ahead of the vehicle; employing light reflected by one or morefiducial markers on the surface onto pixels of a receiver to determine aspatial arrangement of the fiducial markers on the surface; comparingthe spatial arrangement of the fiducial markers with a predetermined mapof the fiducial markers to determine a location of the vehicle; anddetecting a reflection of the stripe from the portion of the surfacewith the receiver, wherein the stripe's reflection is employed todetermine three-dimensional spatial information and visual contrastinformation that is added in real time to the predetermined map for theportion of the surface that is directly ahead of the vehicle.
 22. Thenon-transitory processor readable storage media of claim 21, whereinemploying the scanner comprises employing the scanner to sequentiallyscan the light beam along a line or region of the surface.
 23. Thenon-transitory processor readable storage media of claim 21, wherein thescanner is configured to simultaneously illuminate a line across thesurface with the light beam, wherein employing the scanner comprisesemploying the scanner to sequentially scan a series of lines on thesurface.
 24. The non-transitory processor readable storage media ofclaim 21, wherein each of the fiducial markers has one of a plurality ofdifferent colors, wherein employing light reflected by one or morefiducial markers comprises determining a color of each of the one ormore fiducial markers using the reflected light.
 25. The non-transitoryprocessor readable storage media of claim 21, wherein the actionsfurther comprise employing light reflected from an additional object onthe surface onto pixels of the receiver to determine a position of, orto identify, the additional object.
 26. The non-transitory processorreadable storage media of claim 25, wherein the actions further comprisedetermining a path to avoid the additional object.
 27. Thenon-transitory processor readable storage media of claim 21, wherein theactions further comprise employing light reflected from the surface ontopixels of the receiver to determine condition of the surface.
 28. Thenon-transitory processor readable storage media of claim 27, wherein theactions further comprise adjusting operation of the vehicle in view ofthe determined condition of the surface.
 29. The non-transitoryprocessor readable storage media of claim 27, wherein the actionsfurther comprise adjusting a suspension system in view of the determinedcondition of the surface.
 30. The non-transitory processor readablestorage media of claim 21, wherein employing light comprisessequentially observing subsets of the pixels of the receiver incoordination with the scanning of the light beam.